1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3177 Commits

Author SHA1 Message Date
David Steele
099281c510 Reduce buffer used when size limit set.
If the size is reduced then the amount used should be reduced as well.

This is a bug but does not affect current use cases.
2020-10-07 18:02:05 -04:00
David Steele
39aca3269c Add bufSizeAlloc(). 2020-10-07 16:34:13 -04:00
David Steele
c8a94d9eb6 Add missing assertion in lstFind(). 2020-10-07 15:17:16 -04:00
David Steele
98f6a1cffd Remove unused cfgDefOptionPrefix() function and data.
This function was made obsolete when Perl was removed in f0ef73db.
2020-10-07 12:15:50 -04:00
David Steele
434282d343 Declare cfgDefOptionAllowListValue/Total() static.
These have not been used externally since Perl was removed in f0ef73db.
2020-10-07 12:03:23 -04:00
David Steele
9377d05072
Add repo-azure-endpoint option.
This option allows alternate endpoints (e.g. Azure Government) to be configured.
2020-10-06 17:15:48 -04:00
David Steele
3ce6b3e505 Update config.guess and config.sub to latest versions. 2020-10-05 13:12:48 -04:00
David Steele
3c8774fbab Begin v2.31 development. 2020-10-05 12:02:31 -04:00
David Steele
9229d6c866 v2.30: PostgreSQL 13 Support
Bug Fixes:

* Error with hints when backup user cannot read pg_settings. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Mohamed Insaf K.)

Features:

* PostgreSQL 13 support. (Reviewed by Cynthia Shang.)

Improvements:

* Improve PostgreSQL version identification. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve working directory error message. (Reviewed by Stefan Fercot.)
* Add hint about starting the stanza when WAL segment not found. (Contributed by David Christensen. Reviewed by David Steele.)
* Add hint for protocol version mismatch. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)

Documentation Improvements:

* Add note that pgBackRest versions must match when running remotely. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
* Move info command text to the reference and link to user guide. (Reviewed by Cynthia Shang. Suggested by Christophe Courtois.)
* Update yum repository path for CentOS/RHEL user guide. (Contributed by Heath Lord. Reviewed by David Steele.)
2020-10-05 11:51:45 -04:00
David Steele
74151d57ba Add apt-get update to contributing guide.
This allows apt-get to work correctly even if the base container is a bit out of date.
2020-10-05 08:42:18 -04:00
David Steele
293add28fd Add text rendering for page links missed in 597739fa.
Since links are not possible in the command line help just display the name of the linked section.

Also, during reference text rendering there is no out key so make sure it is defined before trying to use it.
2020-10-05 08:37:48 -04:00
David Steele
597739fafe
Move info command text to the reference and link to user guide.
This means the same text will appear in both places, which should make it easier to find.

Also update the link code to allow both page and section to be specified rather than only one or the other.
2020-09-25 11:26:27 -04:00
David Steele
31962377d9
Add hint and documentation for protocol version mismatches.
Update the documentation to explicitly state that versions must match across hosts when running remotely.

Add a hint to the protocol version mismatch error to help the user identify the problem.
2020-09-25 10:30:29 -04:00
David Steele
b096a25b49 Update test containers for PostgreSQL 13.
Add older PostgreSQL versions to the u18 container that were not available before.

This also updates all minor versions for prior versions of PostgreSQL.
2020-09-24 11:19:51 -04:00
Cynthia Shang
ad79932ba5
Add internal verify command.
Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet).

The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Heath Lord
3e8a6febe6
Update yum repository path for CentOS/RHEL user guide.
Update to the current yum.p.o repository so the CentOS/RHEL user guide will build on PostgreSQL 13.
2020-09-21 14:37:07 -04:00
David Steele
927d9adbee
Improve PostgreSQL version identification.
Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL.

Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required.

Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
2020-09-18 16:55:26 -04:00
David Steele
94475bfbe6 Add missing semicolon in backtrace debug code.
Apparently backtrace has not been used for debugging since it was broken in 7fba1f0b.

Even though this is test code it might be good to find a way to test it to prevent regressions.
2020-09-17 15:17:13 -04:00
David Steele
4cd61152f5 Update PostgreSQL 13 test catalog versions missed in 6bb111c1.
These values are not used by the Perl integration tests so maybe it would be better to remove them, but for now just update since they should not be changing again for PG13.
2020-09-17 12:39:30 -04:00
David Steele
9db3143973
Error with hints when backup user cannot read pg_settings.
This condition used to give a not-very-clear error which we have been intending to improve. But in the meantime the changes in fbff299 resulted in a segfault for this condition instead because the data_directory was assumed to be non-NULL.

Fix this by explicitly throwing an error with hints when any row in pg_settings cannot be selected.
2020-09-17 10:35:27 -04:00
David Steele
c71609879b Fix incorrect control/catalog versions in command/stanza test.
These were not caught because backup info load automatically corrects them.
2020-09-16 09:33:37 -04:00
David Steele
8dce7bbb60
Ignore backup_manifest in PG13.
This file is created by pg_basebackup so might be in the data directory if the cluster was restored from a pg_basebackup backup. Also exclude backup_manifest.tmp since it is possible to find that in the backup directory.
2020-09-14 10:15:40 -04:00
David Steele
fc77c51182
Improve working directory error message.
Improve the wording of the error message and add a hint to make it clearer what is wrong and how the user can fix it.

Also change the assert to a regular error since this is not an internal error.
2020-09-11 10:10:25 -04:00
David Steele
dac182c06e Improve alignment in TEST_ERROR output.
Align the error types and messages so they are easier to compare.
2020-09-11 08:23:26 -04:00
Cynthia Shang
ecb112f802 Add infoArchiveMove() and infoBackupMove(). 2020-09-08 13:02:13 -04:00
Cynthia Shang
5941f5631a Extern infoBackupNewLoad() and infoArchiveNewLoad(). 2020-09-08 12:55:56 -04:00
Cynthia Shang
4239f8a81a Add manifestFree(). 2020-09-08 12:47:35 -04:00
Cynthia Shang
fead004691 Move WAL constants to postgres/interface header. 2020-09-08 12:44:08 -04:00
Cynthia Shang
b8efb13bcb Move archiveIdComparator() to archive/common module. 2020-09-08 12:28:56 -04:00
David Christensen
9fd31913a8 Add hint about starting the stanza when WAL segment not found.
If a stop command has been issued the check command fails due to archiving timing out.

Provide a hint to document this situation and point the user in the proper direction.
2020-09-03 07:49:49 -04:00
David Steele
76900d6764 Fix contributor name. 2020-09-02 14:45:09 -04:00
David Steele
41118e1b68
Allow ProtocolParallel to complete with no jobs.
If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results.

Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone().

For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
David Steele
ce6c492620 Add notes about optional symlinks to repo format 6 design. 2020-09-01 11:02:44 -04:00
David Steele
bb87d989b8 Update config.guess and config.sub to latest versions. 2020-08-31 08:20:40 -04:00
David Steele
05ef03feb5 Begin v2.30 development. 2020-08-31 07:41:48 -04:00
David Steele
1df7d3eee5 v2.29: Auto S3 Credentials on AWS
Bug Fixes:

* Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)
* Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)

Features:

* Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, David Youatt, Aleš Zelený, Jeanette Bromage.)
* Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)

Improvements:

* PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
* Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Brad Nicholson, Oscar. Suggested by Oscar, Brad Nicholson.)
* Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)
* Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)
* Improve detailed error output. (Reviewed by Cynthia Shang.)
* Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)

Documentation Bug Fixes:

* Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)
* Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
* Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)

Documentation Improvements:

* Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)
2020-08-31 07:28:22 -04:00
David Steele
14e1fd10ca Add none to compress-type option reference and fix example. 2020-08-27 10:59:04 -04:00
David Steele
8c2960fab3
Add archive-mode option to disable archiving on restore.
When restoring a cluster that will be promoted but is not intended to be the new primary, it is important to disable archiving to avoid polluting the repository with useless WAL. This option makes disabling archiving a bit easier.
2020-08-25 15:05:41 -04:00
David Steele
1812725c8e Fix typo. 2020-08-25 12:50:06 -04:00
David Steele
851f2e814e
Automatically retrieve temporary S3 credentials on AWS instances.
Automatically retrieve the role and temporary credentials for S3 when the AWS instance is associated with an IAM role. Credentials are automatically updated when they are <= 5 minutes from expiring.

Basic configuration is to set repo1-s3-key-type=auto. repo1-s3-role can be used to set a specific role, otherwise it will be retrieved automatically.
2020-08-25 10:38:49 -04:00
David Steele
1d2b8ae7bb
Improve detailed error output.
Add more info (command, version, options) to asserts, and errors when debug logging is enabled. This won't cover all cases but might mean we get more info in some circumstances.
2020-08-24 14:51:08 -04:00
David Steele
7fba1f0bbe Consolidate variables into structs in common/stackTrace module.
This makes it easier to determine which variables are used in which parts of the module and reduces the chance of name conflicts while unit testing.
2020-08-21 16:12:44 -04:00
David Steele
2a73bd6cbf Move backTraceState variable inside #ifdef WITH_BACKTRACE.
This variable is only used when WITH_BACKTRACE is defined so there's no need to declare it otherwise.
2020-08-21 11:45:54 -04:00
David Steele
0e8e5881d8 Remove invalid call to stackTraceTestFileLineSet().
When testing the common/stack-trace module it is important not to call this test function since the trace stack is empty and it will cause a buffer under run.

Instead use a macro that is only defined under the correct circumstances and add an assert() to catch future regressions.
2020-08-21 11:43:26 -04:00
David Steele
25ee913932 Rearrange members in StackTraceData struct to save memory.
This arrangement is more efficient on 64-bit systems due to alignment. There is no difference on 32-bit systems.
2020-08-20 15:53:01 -04:00
David Steele
859b8a50fd Remove unused parameter from cmdBegin(). 2020-08-20 14:16:36 -04:00
David Steele
fccca0d716 Refactor option logging into a general-purpose function. 2020-08-20 14:11:40 -04:00
David Steele
959f77cd6a
Add general-purpose statistics collector.
Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome.

Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually.

No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible.

The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
David Steele
53f8e7a9cf
Asynchronous list/remove for S3/Azure storage.
Improve the performance of list/delete operations by using async requests.

It's questionable whether this will have any impact on Azure deletes since they are sent one at a time with little work done in between, but it doesn't hurt to try.
2020-08-20 12:12:21 -04:00
David Steele
9876d7096d Disable s390x builds on Travis-CI (again).
s390x currently is too unreliable to be part of required CI builds. They can sometimes take hours or days to get queued which makes getting commits through our workflow very time-consuming.
2020-08-20 11:24:27 -04:00