1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

687 Commits

Author SHA1 Message Date
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists.
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
David Steele
298b33988c Fix non-compliant formatting for function declarations.
Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
David Steele
b4d434b41c Update stanza-create --force documentation to urge caution when using.
Suggested by Jason O'Donnell.
2018-01-31 17:34:27 -05:00
Cynthia Shang
bacc08792d Ensure latest db-id is selected on when matching archive.info to backup.info.
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).

Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include.
Fixed by Cynthia Shang.
2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl.
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05 Improve speed of C unit tests.
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
2018-01-26 16:41:17 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
2cc9b2287b The archive-push command is now partially coded in C.
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
David Steele
a4c058d070 Add 30 second wait loop to lockAcquire() when fail on no lock enabled.
This should help prevent processes that are shutting down from interfering with processes that are starting up.
2018-01-17 15:03:55 -05:00
David Steele
1f39a34abc Move lock release later in exitSafe().
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
2018-01-17 10:31:21 -05:00
David Steele
7c73f4f6c9 Add command and exit modules. 2018-01-17 09:15:51 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
8f81620b9f Improve error module.
Add functions to convert error codes to C errors and handle system errors.
2018-01-16 13:29:27 -05:00
David Steele
bffc6c49b3 Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work. 2018-01-16 13:23:08 -05:00
David Steele
d633417e5d Add Wait object and time module. 2018-01-16 13:05:00 -05:00
David Steele
721d1127a0 Add RegExp object. 2018-01-16 12:57:23 -05:00
David Steele
d992851456 Rename Perl tests so they don't conflict with their C counterparts. 2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
d8c575801d Add Ini object. 2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff Add Storage object. 2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43 Add KeyValue object. 2017-12-22 22:05:37 -05:00
David Steele
2add6cef95 Add Variant and VariantList objects. 2017-12-22 21:50:27 -05:00
David Steele
c4b30af645 Add Buffer object. 2017-12-22 19:56:01 -05:00
David Steele
5667932063 Improve MemContext module.
Add temporary context blocks and refactor allocation arrays to include allocation size.
2017-12-22 18:54:39 -05:00
David Steele
5ea1263d8e Run valgrind on all C unit tests.
Also refactor code to make valgrind happy.
2017-12-22 18:36:36 -05:00
David Steele
b00c6d86b3 Merge 1.27 release notes. 2017-12-19 20:27:31 -05:00
David Steele
70e38a2bd6 Deploy historical documentation to prior rather than the root directory. 2017-12-19 19:14:16 -05:00
David Steele
9fe265d24c No longer run master branch through CI.
The integration branch will be run through CI and then pushed to master with github status checks.
2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster.
Fixed by Cynthia Shang.
Reported by Stephen Frost.
2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10.
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
2017-12-18 18:51:19 -05:00
David Steele
3cc367185e Add documentation for command-line negation. 2017-12-15 11:12:37 -05:00
David Steele
41f5b2cec2 All clusters in the documentation are initialized with checksums. 2017-12-14 15:44:06 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built.
Contributed by Cynthia Shang.
2017-12-13 11:16:27 -05:00
David Steele
0bfda1a908 Document the relationship between the archive-copy and archive-check options.
Suggested by Markus Nullmeier.
2017-12-11 07:48:30 -05:00
David Steele
58b0893067 Improve archive-copy reference documentation. 2017-12-11 07:31:30 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated.
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore.
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00
David Steele
5eb5d94d65 Allow internal options that do not show up in the documentation.
Used for test options initially but other use cases are on the horizon.
2017-12-05 14:46:52 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
4f1725dc36 Replace cfgCommandTotal()/cfgOptionTotal() functions with constants.
The constants are applicable in more cases and allow the compiler to optimize certain loops more efficiently.
2017-12-05 09:45:51 -05:00
David Steele
59fd92fdd5 Move restore test infrastructure to HostBackup.pm.
Required to test restores on the backup server, a fairly common scenario.

Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-03 17:08:49 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb Add unit tests for the Manifest module.
Also minor changes to Manifest module, mostly for test reproducibility.

Contributed by Cynthia Shang.
2017-11-28 11:44:24 -05:00
David Steele
4a0eb6863d Sync time to prevent build failures when running on VirtualBox. 2017-11-27 22:16:24 -05:00
David Steele
332861c6c9 Only build C binary/library for Perl unit/integration tests. 2017-11-27 21:19:59 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
c5ea53bcf9 Add coding standards document. 2017-11-27 16:02:49 -05:00
David Steele
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl.
This allows certain time-critical commands (like async archive-push) to run more quickly.
2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18 Add List and StringList objects to simplify list handling. 2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b Add String object to simplify string handling. 2017-11-26 17:49:10 -05:00
David Steele
7cf955425e The C library is now required.
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
c164fc5fb9 Add memGrowRaw() to memory context module. 2017-11-26 12:22:50 -05:00
David Steele
c360ca5a08 Split "refactor" sections into "improvements" and "development" in the release notes.
Many development notes are not relevant to users and simply clutter the release notes, so they are no longer shown on the website.
2017-11-25 20:32:35 -05:00
David Steele
13893a9731 Relax permissions set by release.pl. 2017-11-21 19:52:56 -05:00
David Steele
ba6b49ecb7 v1.26: Repository Encryption
Bug Fixes:

* Fixed an issue that could cause copying large manifests to fail during restore. (Reported by Craig A. James.)
* Fixed incorrect WAL offset for 32-bit architectures. (Fixed by Javier Wilson.)
* Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. (Fixed by Cynthia Shang. Reported by Clinton Adams.)
* Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
* Fixed an issue passing parameters to remotes. When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed. (Reported by Uspen.)

Features:

* Repository encryption support. (Contributed by Cynthia Shang, David Steele.)
2017-11-21 18:31:54 -05:00
David Steele
033dbcdc20 Update release.pl to push data to site repository. 2017-11-21 17:59:30 -05:00
David Steele
ac54aac448 Fixed an issue passing parameters to remotes.
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
2017-11-21 13:17:44 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function.
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0.
The filter was used with compress level set to 0 which added overhead without any benefit.
2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore.
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 12:46:35 -05:00
David Steele
8ccfaa9ad3 Designate a single distro (Ubuntu 16.04) for C coverage testing
Enable optimizations on the distros that don't do coverage testing.
2017-11-20 09:05:55 -05:00
David Steele
deae0b8cd7 Refactor C page checksum unit test to compile with -Wstrict-aliasing. 2017-11-20 08:58:47 -05:00
David Steele
5d741ea9a9 Refactor C unit test macros so they compile with -Wstrict-aliasing. 2017-11-19 16:46:31 -05:00
David Steele
3b43047e0d Disable -Wclobber compiler warning.
...because it is mostly useless but keep the rest of of -Wextra.
2017-11-19 16:44:33 -05:00
David Steele
dc1a5c18ac Wrap try in a do...while loop to make sure that no random else is attached to the main if block. 2017-11-19 16:30:23 -05:00
David Steele
9395ad7043 Improve base64 implementation.
Different encoded strings could be generated based on compiler optimizations. Even though decoding was still successful the encoded strings did not match the standard.
2017-11-18 22:49:43 -05:00
David Steele
500d6b4b66 Automated tests for 32-bit i386/i686 architecture. 2017-11-18 20:14:14 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files.
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-18 20:02:54 -05:00
David Steele
b701cd5a5f Designate a single distro (Ubuntu 16.04) for Perl coverage testing.
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions.
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00
David Steele
550d4fb9c5 Fix incorrect WAL offset for 32-bit architectures.
Fixed by Javier Wilson.
2017-11-16 16:53:49 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test.
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
2017-11-15 13:17:57 -05:00
Cynthia Shang
bc8543494a Add template to improve initial information gathered for issue submissions.
Contributed by Cynthia Shang.
2017-11-15 11:59:27 -05:00
David Steele
22fe684b1f Only check expect logs on CentOS 7.
Variations in distros cause false negatives in tests but don't add much value.
2017-11-14 17:16:39 -05:00
David Steele
3ee269c099 Add encryption to features list. 2017-11-14 17:07:22 -05:00
David Steele
b8746f368d Inflate performance improvement for gzip filter and full unit test coverage. 2017-11-14 15:12:31 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore.
Reported by Craig A. James.
2017-11-14 14:48:44 -05:00
David Steele
cf7da546a3 Simplify try..catch..finally names. 2017-11-13 21:22:13 -05:00
David Steele
bcdfc7d0b5 Enable additional warnings for C builds. 2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a Repository encryption support.
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
David Steele
f07d2a3d0d Add eof to S3 file driver (required for encryption support). 2017-11-06 12:40:54 -05:00
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
Cynthia Shang
2be93036f1 Clarify usage of the archive-timeout option.
Also describe how it is distinct from the PostgreSQL archive_timeout setting.

Contributed by Cynthia Shang.
Suggested by Keith Fiske.
2017-10-30 14:18:05 -04:00
David Steele
55f68ef013 Remove configurable option hints.
db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.
2017-10-30 10:58:04 -04:00
David Steele
1ef27ec8c2 Add list type for options.
The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.
2017-10-30 10:50:35 -04:00
David Steele
f57e376c44 Update Debian/Ubuntu containers to download lastest version of pip. 2017-10-30 09:29:46 -04:00
David Steele
85b35f30a4 v1.25: S3 Performance Improvements
Bug Fixes:

* Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)
* Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)
* Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reported by Jason O'Donnell.)

Features:

* Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)
2017-10-24 17:36:21 -04:00
David Steele
8674a4f7ae Allow functions with sensitive options to be logged at debug level with redactions.
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
2017-10-24 12:35:36 -04:00
David Steele
b2cc65d886 Improve performance of test code.
Wait when all tests have been assigned to reduce CPU load.
2017-10-24 10:35:42 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored.
Reported by Jens Wilke.
2017-10-22 18:05:46 -04:00
David Steele
3553ccae44 Add I/O performance tests. 2017-10-22 13:27:47 -04:00
David Steele
9b98c2e7d6 Run all tests on tempfs rather than local disk. 2017-10-22 13:11:03 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
44c7c0475a Improve protocol error handling.
In particular, "stop" errors are no longer reported as "unexpected".
2017-10-21 14:11:33 -04:00
David Steele
1f120f3fce Improve performance of list requests on S3.
Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting.

Suggested by Mihail Shvein.
2017-10-20 14:10:16 -04:00
David Steele
f4524aeaa9 Add base64 encode/decode. 2017-10-18 11:14:26 -04:00