1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

3563 Commits

Author SHA1 Message Date
David Steele
131ac0ab5e Rename pckReadNew()/pckWriteNew() to pckReadNewIo()/pckWriteNewIo().
These names more accurately describe the purpose of the constructors.
2021-09-22 11:18:12 -04:00
David Steele
0e76ccb5b7 Convert filter param/result to Pack type.
The Pack type is more compact and flexible than the Variant type. The Pack type also allows binary data to be stored, which is useful for transferring the passphrase in the CipherBlock filter.

The primary purpose is to allow more (and more complex) result data to be returned efficiently from the PageChecksum filter. For now the PageChecksum filter still returns the original Variant. Converting the result data will be the subject of a future commit.

Also convert filter types to StringId.
2021-09-22 10:48:21 -04:00
David Steele
802373cb9d Limit valgrind error output to the first error.
Generally the first error is the only important error. The rest simply lead to a lot of scrolling.
2021-09-21 10:16:16 -04:00
David Steele
473afce57b Copy data page before verifying checksum.
Using UNCONSTIFY() is potentially dangerous since the buffer is modified while calculating the checksum, even though the page is reverted to the original state. Instead make a copy to ensure that the original data is never modified.

This requires the logic to be shuffled a bit since the copy cannot be made until we are sure the page is complete.
2021-09-20 11:03:50 -04:00
David Steele
0efb8adb94 Automatically include all PostgreSQL version interface files. 2021-09-19 20:38:51 -04:00
David Steele
95d814cf81 Specify size for helpData array. 2021-09-19 20:32:27 -04:00
David Steele
912a498b0b Skip comments when rendering help output.
Comments should not appear in the help. They are simply notes on implementation.
2021-09-11 16:07:59 -04:00
David Steele
c38d6926d6 Revert Azurite version for testing to 3.14.0.
3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.
2021-09-09 08:48:45 -04:00
David Steele
f4e1babf6b Migrate command-line help generation to C.
Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.

Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.

The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.

Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
2021-09-08 18:16:06 -04:00
David Steele
def7d513cd Eliminate linefeed formatting from documentation.
Linefeeds were originally used in the place of <p> tags to denote a paragraph. While much of the linefeed usage has been replaced over time, there were many places where it was still being used, especially in reference.xml. This made it difficult to get consistent formatting across different output types. In particular there were formatting issues in the command-line help because it is harder to audit than HTML or PDF.

Replace linefeed formatting with proper <p> tags to make formatting more consistent.

Remove double spaces in all text where <p> tags were added since it does not add churn.

Update all <ul>/<ol>/<li> tags to the more general <list>/<list-item> tags.

Add a few missing periods.
2021-09-08 17:35:45 -04:00
David Steele
245b0c048a Add initializer braces required by older versions of gcc. 2021-09-08 13:32:13 -04:00
David Steele
38eeac48d6 Remove invalid comment block. 2021-09-03 10:39:30 -04:00
David Steele
1afea449e2 Build newer valgrind and move CA cert install in test containers.
The newer version of valgrind helps with some arm64 issues that have been fixed since the architecture has become more popular. Also add the valgrind builds to the Vagrantfile and Dockerfile.

Move the CA cert install from the base container to the test container. This means the CA cert can be changed without rebuilding all the base containers.
2021-09-02 13:26:21 -04:00
David Steele
475b57c89b Allow additional memory to be allocated with a mem context.
The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.

Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.

Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.

Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
David Steele
02b06aa495
Increase max index allowed for pg/repo options to 256.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.

Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.

Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.

Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.

cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 12:09:50 -04:00
John W. Terrell
8a4063c2b6
Add required header for Illumos. 2021-08-27 06:41:56 -04:00
David Steele
5d857e196a Do not use certs in container for testing.
The certs are available in test/certificate so it makes more sense to use them there. In addition the container does not need to be rebuilt unless the CA cert changes.
2021-08-25 07:33:31 -04:00
David Steele
7b4d69211e Add missing header. 2021-08-23 16:09:14 -04:00
David Steele
beafdbfc18 Add const to memContextFreeing()/memContextName() parameters.
These functions do not modify the mem context so they can use const pointers.
2021-08-23 08:37:57 -04:00
David Steele
726d04f13a Remove useless if in memContextFree().
If the upper bound of the loop is 0 then nothing will be done, making the if condition extraneous.
2021-08-23 08:31:50 -04:00
David Steele
d72d4415a7 Use contextParentIdx instead of a loop when moving a context.
contextParentIdx was introduced in 90709dfd to improve the performance of mem context frees. memContextMove() did not get the message, however, and continued to use a loop to find the mem context in the old parent.

Use contextParentIdx to find the mem context in the old parent to avoid a loop.
2021-08-23 08:27:07 -04:00
David Steele
8a9887f27c Remove useless memFree() in regExpNew().
The MEM_CONTEXT_NEW*() block will automatically free memory if there is an error.

This code is pretty old so this may not have been true when it was written.
2021-08-23 07:58:59 -04:00
David Steele
5ffb7cc771 Update config.guess and config.sub to latest versions. 2021-08-23 07:14:22 -04:00
David Steele
aee0e7bac7 Begin v2.36 development. 2021-08-23 07:03:40 -04:00
David Steele
3787cf7803 v2.35: Binary Protocol
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.

Bug Fixes:

* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)

Improvements:

* Binary protocol. (Reviewed by Cynthia Shang.)
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)

Documentation Bug Fixes:

* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)

Documentation Improvements:

* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)

Test Suite Improvements:

* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-08-23 06:52:51 -04:00
David Steele
ad5e063990 Update containers for new PostgreSQL releases.
Also add a package that is required for older Debian versions.
2021-08-19 17:25:57 -04:00
David Steele
590e759968 Rename stringz.h to stringZ.h.
This is more consistent with our file naming conventions.
2021-08-19 11:57:37 -04:00
David Steele
0354144c90 Add instructions for dev builds to stress test in user guide. 2021-08-19 11:35:49 -04:00
David Steele
9b45df7057 Rearrange MemContext struct members to save space on 64-bit.
On 64-bit systems this saves a bit of space (10%) due to alignment. 32-bit systems see no benefit.

Also add tests for the individual struct sizes.
2021-08-19 11:12:56 -04:00
David Steele
bab7a01f99 Mark contextTop static.
This variable is not used outside of the memContext module so no need for it to be extern'd.
2021-08-19 11:00:17 -04:00
David Steele
bb1ca1d309 Change ASSERT() to CHECK() in infoLoad().
Coverity complains that "Argument loaded of ASSERT() has a side effect because the variable is volatile. The containing function might work differently in a non-debug build."

It does not look like this is a real issue, but a CHECK() here is not too expensive for production so change it to silence Coverity.

Also fix a typo in the comment.
2021-08-18 13:32:16 -04:00
David Steele
9ddfcd6a6d Remove dead loop in verifyArchive().
This loop has been dead since the code was initially committed in ad79932b. It looks like it was used at one point but became dead when the enclosing if-else was added during development.

Found by Coverity.
2021-08-18 13:23:24 -04:00
David Steele
56ad427653 Add variables for build paths in the user guide.
This eliminates repetition of the build path so it can be changed more easily.

Also create the build path explicitly rather than suggest that the user do it.
2021-08-18 10:52:35 -04:00
David Steele
944a9e8ff1 Reduce memory requirements of stress test section in the user guide.
The standby memory was set to 1024mb in 86a651f9 to compensate for a memory leak in restore. The leak has been fixed (or at least mitigated) in e1e6e475 and 4fb6384f so the memory can be reduced to 512mb, the same as the primary.
2021-08-18 10:17:10 -04:00
David Steele
d69a596358 Fix infinite loop in protocolServerProcess() on error.
The error was written to the client and then another command read. If the write did not fail then the loop would never exit.

Instead exit on any error that is not raised by the command handler as we can pretty safely assume this is an unrecoverable protocol error. The command handler might throw a protocol error itself, but this should be caught in the next read or write in the main loop.
2021-08-18 09:48:37 -04:00
David Steele
9a6afe3fc1 Fix EOF behavior of ioReadSmall() to match ioRead().
If the buffer was not full at EOF then ioReadSmall() would get stuck in an infinite loop. Instead, return on EOF even if the buffer is not full.

This is not an issue in released versions since ioReadSmall() is not being used.

Also fix a comment typo.
2021-08-18 09:32:20 -04:00
David Steele
4fb6384f10 Fix more memory leaks introduced by the binary protocol in 6a1c0337.
Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
David Steele
e1e6e47535 Add temp mem context reset blocks in job processing loops.
Restore definitely needed to be doing cleanup, just as backup does. The archive-get, archive-push, and verify loop did not seem to be a significant source of leaks but that could change in the future so add resets.

Add temp mem context blocks in the job callbacks where they were missing.

Also switch to the prior context when creating a job, if possible, to save a move.
2021-08-18 08:07:41 -04:00
Cynthia Shang
eca2fc6958
Update config/parse test to use standard patterns. 2021-08-12 12:38:07 -04:00
Cynthia Shang
e17865a03a
Update protocol/protocol test to use standard patterns. 2021-08-12 11:57:17 -04:00
David Steele
86a651f9b0 Add optional stress test section to the user guide.
This is intended to provide pre-release stress-testing. Include container memory limits to help check for memory leaks.

Also add parallelism to make for faster builds.
2021-08-12 07:49:59 -04:00
David Steele
a0bdfa436c
Log backup file total and restore size/file total.
The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.

Add size/file total log message to restore since it did not exist before.
2021-08-11 13:39:36 -04:00
David Steele
6ab18dc0fa
Rearrange backup documentation in user guide.
Remove the "Automatic Stop Option" section since it only applies to PostgreSQL <= 9.6, which will soon be EOL. Since we no longer build the user guide for PostgreSQL < 10 this section was no longer being tested. The stop-auto option is still documented in the reference.

Move the "Fast Start Option" to "Quick Start - Perform Backup". This is a commonly-used option so it makes sense to mention it earlier. This also makes the backups run more quickly. In the worst case, backups in "Quick Start - Perform Backup" could take minutes to start

Move the "Archive Timeout" section to "Quick Start - Perform Backup" since it is the last section in "Backup".
2021-08-11 12:59:25 -04:00
David Steele
f716cb6f4f Fix use after free introduced by the binary protocol in 6a1c0337.
The user and group were stored in a temp reset mem context so they could get freed if there were enough files to trigger the reset in storageRemoteInfoList().

Allocate user and group in a mem context provided by the caller to prevent them being freed prematurely.
2021-08-10 14:22:38 -04:00
David Steele
6789ec420e Add additional checks to performance/storage test.
The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".

To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
2021-08-10 10:37:37 -04:00
David Steele
0eae815ef4 Protect against mixed FUNCTION_TEST* and FUNCTION_LOG* macros.
If the macros are mixed then the debug stack may not be cleaned up correctly. Add variables to ensure that the macros cannot be mixed.

Fix cases where the macros were mixed and add one missing semicolon.
2021-08-09 17:29:09 -04:00
David Steele
c8492aac41 Cleanup inherited clients in protocol test harness.
Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.

We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.
2021-08-09 17:11:39 -04:00
Cynthia Shang
71b654fc29
Fix links and update child process example.
Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 16:56:06 -04:00
Cynthia Shang
f653b59664
Update db/db test to use standard patterns. 2021-08-09 16:35:48 -04:00
David Steele
2d5cbfc731 Fix memory leaks introduced by the binary protocol in 6a1c0337.
Additional buffers were being allocated for the protocol messages but not being freed.

Most of the allocations were fairly harness, but storageRemoteOpenReadProtocol() and storageWriteRemote() were problematic because they were allocating (but not freeing) buffers equal to the transfer size of the file. Depending on compression, this could be a lot of memory. Though the memory was freed after each file transfer the aggregate of memory used during parallel processing could overwhelm systems with constrained memory.

Also allocate larger initial buffers in storageRemoteOpenReadProtocol() and storageWriteRemote() so a reallocation is not needed.
2021-08-09 09:28:17 -04:00