1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3888 Commits

Author SHA1 Message Date
David Steele
efbcd975c4 Update Docker test image for Debian 9.
A change invalidated the current image which has been causing the d9 test to run longer.
2022-05-06 13:56:58 -04:00
David Steele
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
David Steele
475e7c692d Clean up dividers in the documentation.
Dividers were used in some files, but not others, and some had section names (which are hard to maintain) and others did not.

Try to make this more consistent by putting a divider on front of every section, variable block, and wherever else seems appropriate.
2022-05-06 12:11:04 -04:00
David Steele
356bc27bf2 Remove key dividers in help.xml.
The idea was to make this file easier to browse and edit, but in fact it is much easier to just search for the command/option needed.

The dividers were never applied consistently and at some point we decided to get rid of the comments because they were hard to keep updated. The result was a mix of styles which did nobody any favors.
2022-05-06 11:41:28 -04:00
Reid Thompson
65d22e4325
Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
David Steele
f405fc6ae2
Backup file bundling documentation.
Make the feature user visible and add documentation to the user guide.
2022-05-06 10:21:20 -04:00
Reid Thompson
4cc0d46d60 Fix comment wrapping. 2022-05-06 09:34:39 -04:00
David Steele
e70c71049e Use uint8_t for optionResolveOrder.
This saves a bit of space and should not affect processing speed.

On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.
2022-05-06 07:49:23 -04:00
David Steele
808f7bf11c Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.
This test was likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:14:13 -04:00
David Steele
efe0a39a75 Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.
These tests were likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:01:02 -04:00
David Steele
5089a26633 Convert strNewFmt() to THROW_FMT() in config/parse module.
It's not clear why strNewFmt() was used here, but there is no need for it.
2022-05-05 18:35:00 -04:00
David Steele
876f3bbd1c Remove COLON_STR and separator parameter from cfgParseCommandRoleName().
The separator parameter in cfgParseCommandRoleName() was useless since it was always set to : and COLON_STR did not provide any clarity its the single other usage.
2022-05-05 18:15:05 -04:00
David Steele
7ae5478d98 Remove most _Z constants.
Most of the time these were not making the code any clearer.

For cases where they were used to construct Strings and Buffers, replace with constants.

Also cleanup unused Buffers and Strings.
2022-05-05 12:09:21 -04:00
David Steele
a6b1adb5fd Remove extraneous linefeed when writing a lock file.
Linefeeds are no longer part of the lock file format.
2022-05-05 11:15:14 -04:00
David Steele
5f8c9cd66a
Add ClockError for unexpected clock skew and timezone changes.
A distinct result code should help debugging of clock skew and timezone issues.
2022-05-05 10:19:11 -04:00
David Steele
b6bfd9f99d
Strip extensions from history manifest before showing in error message.
In cases where clock skew or timezone issues are preventing backup label generation the user could see an error like this:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I.manifest.gz'

This will happen if the most recent label is drawn from the history. It is cleaner (and probably less confusing) to strip off the extensions so the user sees:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I'
2022-05-05 09:20:49 -04:00
David Steele
ef672c74ad
Prevent memContextFree() from freeing memory needed by callbacks.
The order of callbacks and frees meant that memory needed during a callback (for logging in all known cases) might end up being freed before a callback needed it.

Requiring callbacks and logging to check the validity of their allocations is pretty risky and it is not clear that all possible cases have been accounted for.

Instead recursively execute all the callbacks first and then come back and recursively free the context. This is safer and it removes the need to check if a context is freeing so a simple active flag (in debug builds) will do. The caller no longer needs this information at all so remove memContextFreeing() and objMemContextFreeing().
2022-05-04 14:53:05 -04:00
Reid Thompson
d9088b2e2b
Show backup percent complete in info output.
In the JSON output the percent complete is storage as an integer of the percent complete * 100. So, before display it should be converted to double and divided by 100, or split using integer mod and div.

Note that percent complete will only be displayed on the host where the backup was executed. Remote hosts will show a backup/expire running with no percent complete.
2022-05-04 12:52:05 -04:00
David Steele
20782c88bc
PostgreSQL 15 support.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.

This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
2022-05-04 11:55:59 -04:00
David Steele
8e849ae85d Add PRs to thread locking on Github actions.
It also makes sense to lock old PRs. They can be manually unlocked if they are needed for some reason.

Also add output logging to make it easier to determine if thread locking is completing.
2022-05-04 10:28:39 -04:00
David Steele
09b387fccd Move issue locking to Github actions.
The old plugin has been defunct for some time so there are currently a lot of unlocked issues.

Running this once per week seems sufficient for now. Worst case it can be run manually if it gets behind.
2022-05-04 09:24:35 -04:00
David Steele
692fe496bd
Remove dependency on pg_database.datlastsysoid.
This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column.

Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available.

The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.
2022-05-04 08:22:45 -04:00
David Steele
302e0c0921 Remove extra linefeed. 2022-05-03 16:53:29 -04:00
David Steele
9a271e925c
Fix error thrown from FINALLY() causing an infinite loop.
Any error thrown resets execution to the last setjmp(), which means that parts of the try block need to make sure they don't get run again. FINALLY() was not doing this so if it threw an error it would end up back in the FINALLY() block, where the error would likely be thrown again, causing an infinite loop.

Fix this by tracking the state of FINALLY() and only running it once. This requires cleaning the error stack like CATCH*() and clearing the error like TRY_END() depending on the order of execution.
2022-05-03 14:34:05 -04:00
David Steele
b89c568b5f Fix obsolete variable naming. 2022-05-03 10:50:48 -04:00
David Steele
9629908694
Error on all lock failures except another process holding the lock.
The archive-get/archive-push commands would not error for, .e.g permissions errors, when attempting to get a lock before launching the async process. Since the async process was not launched there would be no error status file and the user would get a generic failure message. Also, there would be no async log.

Refactor lockAcquireFile() to throw an error when failOnNoLock = false unless the file is locked by another process. This seems to be the original intent of this parameter and there may have been a mistake when porting from Perl. In any case it looks wrong enough to be considered a bug.
2022-05-03 10:13:32 -04:00
David Steele
eb435becb3 Exclude mem context name from production builds.
The mem context name is used to produce clearer debug errors but it has no purpose in production builds.

Also remove memContextName() and access the struct directly since the name is only used within the common/memContext module.

Note that a few errors that were thrown in production builds (and required the name) are now only thrown in debug builds. In practice we have not seen these errors in production builds due to extensive coverage so it does not seem worth modifying the error to work without the context name.

This saves some memory, which is worthwhile, but the goal is to refactor Strings and Variants to have their own mem contexts and this change will prevent them from using more memory than they are now, along with other changes that will be coming later.
2022-05-02 15:17:34 -04:00
David Steele
0055fa40fe Add user:group to lock permission error.
This will help debug permissions errors when the lock file cannot be created.
2022-05-02 09:45:57 -04:00
David Steele
03c71aa606 Add hint to check the log on archive-get/archive-push async error.
If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log.

This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
David Steele
126fc99c77 Fix leaks in common/json, common/keyValue, and common/variantList.
This doesn't solve the problem of the variant code making far too many copies, but it at least plugs the leaks.

jsonReadVarRecurse() could leak KeyValue and VariantList.

kvDup() leaked object allocations into the calling context.

kvDefault() gets a more efficient return structure.

kvGetList() leaked a Variant into the calling context.

varLstNewStrLst() leaked object allocations into the calling context. Update varLstDup() to reflect changes made in varLstNewStrLst().
2022-04-28 14:10:53 -04:00
David Steele
96166539cf Fix leaked String in cfgParseSize(). 2022-04-28 13:33:03 -04:00
David Steele
4872a3f121 Improvements to test harness memory debugging.
Only set -DDEBUG_MEM for the modules currently being tested rather than globally.

Also run tests in a temp mem context. Running in the top context can confuse memory accounting when a new context is created in the top context.
2022-04-28 12:33:39 -04:00
David Steele
90f939b36f Fix leaks in common/io unit test.
These leaks make it harder to detect leaks in the core code, so fix them.
2022-04-28 12:31:59 -04:00
David Steele
8047e97e31 Fix leaked String and Variant in harnessPqScriptRun(). 2022-04-28 12:17:33 -04:00
David Steele
ceb303f9e2 Fix comment typo. 2022-04-28 11:46:55 -04:00
David Steele
c463993b4c Fix leaks in the storage/s3 module.
storageS3Helper() leaked a few Strings which ended up in a long-lived context.

storageS3AuthAuto() and storageS3AuthWebId() were cleaned up by their callers but since they are not called often a temp mem context seems better.

storageS3Request() leaked an HttpRequest.

storageS3Info() leaked an HttpResponse.

storageS3PathRemoveInternal() leaked a variety of objects. Fix by freeing some of them and adding a temp mem context.

storageS3Remove() leaked an HttpResponse object.

storageWriteS3Part() leaked an HttpResponse object.
2022-04-28 11:41:16 -04:00
David Steele
4750bc94dd Fix leaks in the storage/remote module.
storageRemoteFilterGroup() leaked a number of objects. Use a temp mem context to prevent that.

storageRemoteProtocolInfoListCallback() leaked a PackWrite.

storageWriteRemoteFreeResource() leaked a PackWrite.
2022-04-28 11:20:31 -04:00
David Steele
c123a6af9f Fix leaks in the storage/posix module.
storagePosixPathCreate() leaked a String.

storagePosixPathRemoveCallback() leaked a String.
2022-04-28 11:19:01 -04:00
David Steele
d89bc6f2d2 Fix leaks in the storage/gcs module.
storageGcsAuthToken() memory was being cleaned up by the calling context, but seems better to keep this tidy and add a temp mem context.

storageGcsRequest() leaked an HttpRequest.

storageGcsInfo() leaked a number of objects. Use a temp mem context to prevent that.

storageGcsPathRemoveCallback() leaked an HttpResponse.

storageGcsRemove() leaked an HttpReponse.

storageWriteGcsVerify() leaked a number of objects. Use a temp mem context to prevent that.

storageWriteGcsBlock) leaked an HttpReponse.
2022-04-28 10:53:11 -04:00
David Steele
083c93eaa3 Reuse Strings in iniLoad().
Reuse the section/key/value Strings by truncating them instead of creating a new one every time.

Also add an error for empty sections. This function is only used for loading info files (not config files), which should never contain an empty section.
2022-04-28 10:11:15 -04:00
David Steele
bc46d4e37b Add cvtZSubNTo*() functions.
These functions allow conversion from substrings without needing to create a String or a temporary buffer.

httpDateToTime() no longer requires a temp mem context. Also improve handling of month search to avoid an allocation.

httpUriDecode() no longer requires a temp mem context.

jsonReadStr() no longer requires a temp mem context.

pgLsnFromWalSegment() no longer requires a temp mem context.

pgVersionFromStr() no longer requires a temp mem context. Also do a bit of refactoring.

storageGcsCvtTime() no longer leaks six Strings per call.

storageS3CvtTime() no longer leaks six Strings per call.
2022-04-28 09:50:23 -04:00
David Steele
6e18235be8 Fix leak in varLstNewStrLst().
The duplicated list elements were leaked into the calling context.
2022-04-28 06:52:07 -04:00
David Steele
aeb7568a0f Remove unused header. 2022-04-28 06:48:02 -04:00
David Steele
5285b9aa7b Fix incorrect test param. 2022-04-27 20:18:18 -04:00
David Steele
eb65a5674d Use OBJ_NEW*() macros in SocketServer object.
This was missed in ccc255d3 when the TLS server was introduced, probably because work on that commit preceded when the macros were introduced in 475b57c8. It would have been easy to miss in a merge.
2022-04-27 11:40:29 -04:00
David Steele
6f2654a5eb Fix leaks in the storage/azure module.
storageAzureHelper() leaked a few Strings which ended up in a long-lived context.

storageAzureNew() failed to make a copy of the endpoint. This worked because storageAzureHelper() leaked the endpoint into the long-lived parent context.

storageAzureRequest() leaked an HttpRequest.

storageAzureInfo() leaked an HttpResponse.

storageAzurePathRemoveCallback() leaked an HttpResponse.

storageAzureRemove() leaked an HttpResponse.
2022-04-26 12:53:55 -04:00
David Steele
be120c746c Refactor and fix leak in storageRepoPathExpression().
storageRepoPathExpression() could leak a StringList. Also refactor to remove unneeded assignments and create the String one time.
2022-04-26 12:27:51 -04:00
David Steele
cca6df872a Refactor functions in postgres/interface module and fix leak.
pgLsnFromWalSegment() leaked two Strings.

Refactor pgLsnRangeToWalSegmentList() to create the StringList in the calling context rather than moving it later.
2022-04-26 12:09:44 -04:00
David Steele
a56fa0eb45 Fix leaks in protocol module.
These leaks were not a big deal individually and there are generally few protocol objects created, but the leaks ended up in mem contexts that persist for most of execution. It makes sense to keep long-lived contexts as tidy as possible.
2022-04-26 11:59:21 -04:00
David Steele
78e912a932 Fix leaks in info module.
*LoadFileCallback() all leaked between retries.

infoPgArchiveId() leaked a String.
2022-04-26 11:20:51 -04:00