1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

366 Commits

Author SHA1 Message Date
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
55f68ef013 Remove configurable option hints.
db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.
2017-10-30 10:58:04 -04:00
David Steele
1ef27ec8c2 Add list type for options.
The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.
2017-10-30 10:50:35 -04:00
David Steele
85b35f30a4 v1.25: S3 Performance Improvements
Bug Fixes:

* Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)
* Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)
* Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reported by Jason O'Donnell.)

Features:

* Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)
2017-10-24 17:36:21 -04:00
David Steele
8674a4f7ae Allow functions with sensitive options to be logged at debug level with redactions.
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
2017-10-24 12:35:36 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored.
Reported by Jens Wilke.
2017-10-22 18:05:46 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
44c7c0475a Improve protocol error handling.
In particular, "stop" errors are no longer reported as "unexpected".
2017-10-21 14:11:33 -04:00
David Steele
1f120f3fce Improve performance of list requests on S3.
Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting.

Suggested by Mihail Shvein.
2017-10-20 14:10:16 -04:00
Cynthia Shang
2d56de00fc Fix instances where database-id was not rendered as an integer in JSON info output.
Fixed by Cynthia Shang.
Reported by Jason O'Donnell.
2017-10-18 08:42:32 -04:00
David Steele
c94e52f697 Remove error when overlapping timelines are detected.
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.

Reported by blogh.
2017-10-16 14:34:44 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler.
Basic try ... catch ... finally pattern to simplify error handling in C.
2017-10-16 10:09:56 -04:00
David Steele
8efcc38304 Improvements in C codebase:
* Update C naming conventions.
* Use int datatype wherever possible.
* Better separation of C source from Perl interface.
2017-09-30 10:44:03 -04:00
David Steele
d0bf946857 v1.24: New Backup Exclusions
Bug Fixes:

* Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. (Reported by Uspen.)
* Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. (Reported by Uspen.)

Features:

* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-28 19:06:41 -04:00
David Steele
4d6c56b4d7 Open log file after async process is completely separated from the main process.
This prevents the main process from also logging to the file.

Suggested by Jens Wilke.
2017-09-20 08:08:36 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
6343fdd584 Additional backup exclusions.
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 08:26:57 -04:00
David Steele
df1ec0aaeb v1.23: Multiple Standbys and PostgreSQL 10 Support
Bug Fixes:

* Fixed an issue that could cause compression to abort on growing files. (Reported by Jesper St John, Aleksandr Rogozin.)
* Fixed an issue with keep-alives not being sent to the remote from the local process. (Reported by William Cox.)

Features:

* Up to seven standbys can be configured for backup from standby. (Contributed by Cynthia Shang.)
* PostgreSQL 10 support.
* Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. (Suggested by Victor Gdalevich.)

Refactoring:

* Configuration rules are now pulled from the C library when present.
* Increase HTTP timeout for S3.
* Add HTTP retries to harden against transient S3 network errors.
2017-09-03 17:42:00 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
80e5e8db01 Increase HTTP timeout for S3. 2017-09-01 19:16:57 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways.
Suggested by Victor Gdalevich.
2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.`
Contributed by Cynthia Shang.
2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files.
Reported by Jesper St John, Aleksandr Rogozin.
2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
84caca0f34 v1.22: Fixed S3 Retry
Bug Fixes:

* Fixed authentication issue in S3 retry.
2017-08-09 11:50:02 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
2017-08-08 18:09:07 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade.
Fixed by Cynthia Shang.
2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
df7d633683 Improvements to command/command-line help:
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups.
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
33d769e147 v1.20: Critical 8.3/8.4 Bug Fix
Bug Fixes:

* Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
* Fixed missing flag in C library build that resulted in a mismatched binary on 32-bit systems. (Reported by Adrian Vondendriesch.)

Features:

* Add s3-repo-ca-path and s3-repo-ca-file options to accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs. (Suggested by Scott Frazer.)

Refactoring:

* Harden protocol handshake to handle race conditions.
* Fixed misleading error message when a file was opened for write in a missing directory.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
* Rename Archive modules to remove redundancy.
* Improve S3 error reporting.
* Minor optimizations to package loads and ordering for archive-get and archive-push commands.
2017-06-27 22:42:31 -04:00
David Steele
9e248a6627 Minor optimizations to package loads and ordering for archive-get and archive-push commands. 2017-06-27 17:54:30 -04:00
David Steele
f3b62d2d67 Fixed misleading error message when a file was opened for write in a missing directory. 2017-06-27 17:07:12 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
f596702c5b Improve S3 error reporting. 2017-06-21 20:46:49 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor.
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
2017-06-15 15:32:10 -04:00