Warn if a global variable is defined without a previous declaration. Use this option to detect global variables that do not have a matching extern declaration in a header file.
This code was duplicated in each driver so this means less duplication.
In addition, some drivers were not creating a parameter list for decompression which meant they could not be used remotely. This is not a currently a bug since none of them were being used remotely, but it was a blocker for using lz4 for protocol compression.
The integration tests could fail if:
1. After restoring the PostgreSQL instance the recovery process starts, which calls asynchronous archive-get.
2. After archive-get checks the existence of the queue directory, but before writing the WAL file, there are restores when the next test is begun, which leads to the deletion of the queue directory.
3. Since the directory no longer exists, writing the WAL file will fail, and archive-get will write the error file to the queue.
4. A new PostgreSQL instance will start and the recovery process will begin, which requests the WAL file.
5. The new archive-get looks into the queue directory, finds the error file, and throws out the error, after which the PostgreSQL recovery fails because the previous archive-get background process has not finished yet.
This patch fixes the problem by using a separate spool directory for each test.
An in 355e27d6, it makes sense to exclude FUNCTION_(LOG|TEST)_RETURN_VOID() macros when then they are on the last line of a function because in this case they are a noop (but are still used for debugging).
8d6bceb5 refactored version/help to operate more like regular commands in part to simplify the implementation of --version and --help. Unfortunately this had the side effect of these commands also loading pgbackrest.conf which would lead to an error if the file could not be read or parsed.
Add a filter to prevent version or help from loading pgbackrest.conf. Also prevent reads from the env to stop any warnings or errors from that source.
0c32757f made lz4 required in the meson build but conditional compilation was left in to make reverting easy for packagers just in case.
Since a few releases have gone by without any complaints, remove conditional compilation for lz4.
Warn if anything is declared more than once in the same scope, even when the extra declaration is valid and changes nothing. This is primarily useful for catching missing header ifdef barriers.
Move the environ variable into config/parse.h since it must be declared by us and we use it multiple times.
Warn if a global function is defined without a previous prototype declaration. This helps detect when a function that should be static is accidentally declared extern.
Most of the changes are to add missing header files so functions can see their declarations.
In a some cases functions that should have been static were marked as such. There were only five of these in the core but every little bit counts.
Lastly, it was necessary to suppress the warning in the postgres test modules where the function declarations are not available. This is fixable by aligning the module with the auto-generated code in core, but is not a priority.
Warn whenever a pointer is cast so as to remove a type qualifier from the target type. For example, warn if a const char * is cast to an ordinary char *.
Most of the changes for this are fairly rote: just add a const qualifier where needed. In some cases functions needed to be reworked to return non-const where before they had returned const and then cast it back to non-const. None of these patterns appeared to be bugs, but they were certainly misleading.
Some cases (especially excvp() and calls to bz2) could not be fixed because of how functions out of our control are defined. In those cases the warnings have been suppressed and a comment added to detail the exception. This was also done a few places in the tests.
These three objects can be created as constants at compile time using specialized macros. Unfortunately since the values assigned are also const, cast-qual complained about the cost qualifier being lost.
Fix this by creating new structures to be used just for creating these constants. This is not ideal due to the need to keep the duplicated structures in sync, but in practice these structures are almost never modified. Testing should catch any out of sync structures and this feature is valuable enough to keep even though in theory there could be memory safety issues. In practice the APIs prevent const objects from being used in an unsafe way and testing provides a fair assurance of safety. Writing to these consts would be a fatal error even if it did not cause a segfault.
Ideally, we would be able to use warning suppression in these macros to avoid the extra struct, but due to the way they are used it is not possible to add the required pragmas (even using _Pragma).
Finally this construction makes it obvious that something special is being done, rather than it being under the covers.
The volatile qualifiers should be on the pointers that are stored on the stack, not on the structures that are allocated by OpenSSL functions.
Also remove all the casts that were required when volatile was in wrong place.
Per our policy to support five EOL versions of PostgreSQL, 9.4 is no longer supported by pgBackRest. Remove all logic associated with 9.4 and update the tests.
This includes a small fix in infoPg.c to allow backup.info files with old versions to be saved. This allows expire to function when old versions are present. Even though those older versions cannot be used, they can be expired.
Tests for 9.4 are left in the expire/info tests to demonstrate that these commands work with old versions present.
NOTE TO PACKAGERS: This is last feature release to support the autoconf/make build. Please migrate to meson if you have not already done so. 2.54.X patch releases (if any) will continue to support autoconf/make.
Bug Fixes:
* Fix PostgreSQL query performance for large datasets. (Fixed by Thibault Vincent, David Steele. Reviewed by David Christensen, Antoine Millet. Reported by Antoine Millet.)
Features:
* Allow repositories on versioned storage to be read at a target time. (Reviewed by Stefan Fercot, David Christensen.)
* Allow requested standby backup to proceed with no standby. (Reviewed by Stefan Fercot.)
Improvements:
* Summarize backup reference list for info command text output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Refresh web-id token for each S3 authentication. (Contributed by Brent Graveland. Reviewed by David Steele.)
* Correctly display current values for indexed options in help. (Reviewed by David Christensen.)
* Save backup.info only when contents have changed. (Reviewed by Stefan Fercot.)
* Remove limitation on reading files in parallel during restore. (Reviewed by David Christensen.)
* Improve SFTP error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
Documentation Features:
* Add performance tuning section to user guide. (Reviewed by Stefan Fercot.)
Documentation Improvements:
* Clarify source for data_directory. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Matthias.)
* Better logic for deciding when a summary should be lower-cased. (Suggested by Daniel Westermann.)
The action suddenly broken and this appears to be the best work around according to https://github.com/uraimo/run-on-arch-action/issues/155.
Unfortunately the tests take almost twice and long to run, probably because the container needs to be built from scratch.
There have been occasional SFTP authentication failures on 32-bit. We are planning to drop 32-bit support so it does not seem worth chasing these errors down and they are likely timing issues anyway.
In the case of a rapid restart it is possible that the socket may not be immediately available, so retry until it becomes available.
This is particularly useful for testing where sockets are bound and released very rapidly.
Option help summaries do not have initial capitals (except in special cases) and final periods so it makes sense to render the command summaries the same way.
Use the same function for both so they are consistent.
The asynchronous logic used to implement the query timeout was misusing PQisBusy(), which caused the wait handler to throttle the consumption of command results. It could introduce a large delay on a query up to `db-timeout` because of the back-off sequence.
Following the recommendation of libpq, fix by polling the client socket for data availability and then continue consuming results and checking for command busyness.
This is useful for code that has its own wait mechanism, e.g. poll(), but still needs a way to track overall time elapsed.
To keep it simple waitRemains() is called by waitMore().
Ubuntu 20.04 has been having consistent errors starting PostgreSQL 10 so move 9.5 to this container instead. An older version makes sense with an older distro.
Also move PostgreSQL 12 from RHEL 8 since this version will be EOL soon.
Containers are notoriously unfriendly to systemctl (really systemd) but we prefer to use systemctl to make our documentation as accurate as possible. This replacement performs all the functions of systemctl without requiring systemd, which great simplifies container configuration and allows the documentation build to work in more environments.
The current value for an indexed option was always for the first index, e.g. pg1-path. This is likely legacy from before indexing was added (and faithfully copied over from Perl, apparently).
Fix this by enumerating the current values in the option help and displaying <multi> in the option list when more than one value exists.
Full debug/trace logging in production is unlikely to be useful but does use space in the binary.
Reduce logging to be useful for testing but not be deployed in production.
We frequently tell users to enable to these options but they are spread through the documentation and not at all obvious. Hopefully putting them in the quick start will make them more visible and also provide an easy place to link.
Options that are only valid on the command-line should not appear in the configuration reference because it implies that they can be added to pgbackrest.conf, which is not the case.
Most command-line options were already excluded because they lacked a section, but a few were slipping through.
This feature allows the archive-get, info, repo-get, repo-ls, restore, and verify commands to operate at a point-in-time on versioned buckets in Azure, GCS, and S3. This allows recovery even if a repository has been accidentally or maliciously deleted or corrupted.
This restriction prevented multiple files being read from a remote simultaneously, which was not supported by the protocol. Although the limitation only applied to remotes, it was applied in all cases for testing purposes and because we planned to fix it.
Protocol command multiplexing added in df8cbc91 allows files to be read simultaneously from a remote so this restriction is no longer required.
Note that there is a test for this condition since the prior code had coverage. It might be tricky to ensure that test doesn't go away, but in general we should have enough tests in place to ensure simultaneous reads function as expected.
Add a "prefer" value to the backup-standby option to allow the backup to proceed when no standby is found. Note that this will not help if the standby is responding but fails to sync with the primary after the backup has started.
Introduce a new option modifier, bool-like, that allows a boolean option to be converted to a string or string-id option while still allowing the option to act like a boolean on the command-line, e.g. --no-backup-standby.
This prevents backup.info from being saved again when expire does not make any changes.
More importantly, as we look to support versioning on object stores, it will be much easier to determine a good point-in-time to use for restore if there are no extraneous saves of backup.info.
In these functions infoBackup was marked as const even though it was modified in the function. This was allowed by the compiler because the infoBackup struct was not being directly modified but it still goes against our coding conventions.
The token file pointed to by the AWS_WEB_IDENTITY_TOKEN_FILE environment variable was read once at startup, but for long operations the token might expire before completion.
Instead read the token on each S3 authentication so the current token is always used.
