IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
Bug Fixes:
* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)
Improvements:
* Binary protocol. (Reviewed by Cynthia Shang.)
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)
Documentation Improvements:
* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)
Test Suite Improvements:
* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Coverity complains that "Argument loaded of ASSERT() has a side effect because the variable is volatile. The containing function might work differently in a non-debug build."
It does not look like this is a real issue, but a CHECK() here is not too expensive for production so change it to silence Coverity.
Also fix a typo in the comment.
This loop has been dead since the code was initially committed in ad79932b. It looks like it was used at one point but became dead when the enclosing if-else was added during development.
Found by Coverity.
This eliminates repetition of the build path so it can be changed more easily.
Also create the build path explicitly rather than suggest that the user do it.
The standby memory was set to 1024mb in 86a651f9 to compensate for a memory leak in restore. The leak has been fixed (or at least mitigated) in e1e6e475 and 4fb6384f so the memory can be reduced to 512mb, the same as the primary.
The error was written to the client and then another command read. If the write did not fail then the loop would never exit.
Instead exit on any error that is not raised by the command handler as we can pretty safely assume this is an unrecoverable protocol error. The command handler might throw a protocol error itself, but this should be caught in the next read or write in the main loop.
If the buffer was not full at EOF then ioReadSmall() would get stuck in an infinite loop. Instead, return on EOF even if the buffer is not full.
This is not an issue in released versions since ioReadSmall() is not being used.
Also fix a comment typo.
Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
Restore definitely needed to be doing cleanup, just as backup does. The archive-get, archive-push, and verify loop did not seem to be a significant source of leaks but that could change in the future so add resets.
Add temp mem context blocks in the job callbacks where they were missing.
Also switch to the prior context when creating a job, if possible, to save a move.
This is intended to provide pre-release stress-testing. Include container memory limits to help check for memory leaks.
Also add parallelism to make for faster builds.
The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.
Add size/file total log message to restore since it did not exist before.
Remove the "Automatic Stop Option" section since it only applies to PostgreSQL <= 9.6, which will soon be EOL. Since we no longer build the user guide for PostgreSQL < 10 this section was no longer being tested. The stop-auto option is still documented in the reference.
Move the "Fast Start Option" to "Quick Start - Perform Backup". This is a commonly-used option so it makes sense to mention it earlier. This also makes the backups run more quickly. In the worst case, backups in "Quick Start - Perform Backup" could take minutes to start
Move the "Archive Timeout" section to "Quick Start - Perform Backup" since it is the last section in "Backup".
The user and group were stored in a temp reset mem context so they could get freed if there were enough files to trigger the reset in storageRemoteInfoList().
Allocate user and group in a mem context provided by the caller to prevent them being freed prematurely.
The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".
To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
If the macros are mixed then the debug stack may not be cleaned up correctly. Add variables to ensure that the macros cannot be mixed.
Fix cases where the macros were mixed and add one missing semicolon.
Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.
We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.
Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
Additional buffers were being allocated for the protocol messages but not being freed.
Most of the allocations were fairly harness, but storageRemoteOpenReadProtocol() and storageWriteRemote() were problematic because they were allocating (but not freeing) buffers equal to the transfer size of the file. Depending on compression, this could be a lot of memory. Though the memory was freed after each file transfer the aggregate of memory used during parallel processing could overwhelm systems with constrained memory.
Also allocate larger initial buffers in storageRemoteOpenReadProtocol() and storageWriteRemote() so a reallocation is not needed.
This makes the generated HTML much more readable in diffs because a single word change will not change a line with potentially many tags.
The output is now slightly larger because of the extra linefeeds.
Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.
Also fix an outdated comment.
pg1 was incorrectly used instead of {[host-pg1]} which meant the wrong host name was displayed.
Also, the install block was installing packages to the build host no matter which host was specified.
This function was included in a header but not declared inline, so linker errors happened when the header was included into more than one file.
Because of the setjmp() in TRY_BEGIN() it can't be inlined so put it in a C file.
Also add some missing headers.
There have been intermittent failures on f33 (with coverage) but not on u16 (without coverage).
Reproducing this reliably has been very difficult, so just try increasing the timeouts. This is based on the observation that tests with coverage take longer than tests without, which may lead the f33 tests to fail if CI is running slower than usual.
This will not increase the runtime of the test unless there is an error.
If configure/make has been run in the src path it can conflict with tests, which may require different build options.
Also add a comment when rebuilding for code generation.
This file duplicated the command list that already exists in parse.auto.c.
Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.
Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.
This was started in c5ae047e but did not include generation of parse.auto.c.
The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.
Option order resolution now includes cycle detection.
These defaults were not getting the multiplier so the timeouts were much lower than expected.
Since PostgreSQL retries get/push this was probably not a big deal, but it could be critical in the future for a different time value.
Fix booleans that were set to y rather than true and defaults that were set to true instead of 1. Perl was tolerant of these but C is not.
Reorder repo-storage-verify-tls to satisfy inheritance ordering for the dependency.
Remove strIdGenerate() since bldStrId() performs the same function without cluttering the core code. Since bldStrId() is intended to work in non-debug builds, move the validity checks for input strings out of the DEBUG block.
StringIds are generated as 5/6 bit, whichever is most efficient, for each option value. cfgOptionStrIdInternal() has been updated for this logic.
