Warn whenever a pointer is cast so as to remove a type qualifier from the target type. For example, warn if a const char * is cast to an ordinary char *.
Most of the changes for this are fairly rote: just add a const qualifier where needed. In some cases functions needed to be reworked to return non-const where before they had returned const and then cast it back to non-const. None of these patterns appeared to be bugs, but they were certainly misleading.
Some cases (especially excvp() and calls to bz2) could not be fixed because of how functions out of our control are defined. In those cases the warnings have been suppressed and a comment added to detail the exception. This was also done a few places in the tests.
These three objects can be created as constants at compile time using specialized macros. Unfortunately since the values assigned are also const, cast-qual complained about the cost qualifier being lost.
Fix this by creating new structures to be used just for creating these constants. This is not ideal due to the need to keep the duplicated structures in sync, but in practice these structures are almost never modified. Testing should catch any out of sync structures and this feature is valuable enough to keep even though in theory there could be memory safety issues. In practice the APIs prevent const objects from being used in an unsafe way and testing provides a fair assurance of safety. Writing to these consts would be a fatal error even if it did not cause a segfault.
Ideally, we would be able to use warning suppression in these macros to avoid the extra struct, but due to the way they are used it is not possible to add the required pragmas (even using _Pragma).
Finally this construction makes it obvious that something special is being done, rather than it being under the covers.
Per our policy to support five EOL versions of PostgreSQL, 9.4 is no longer supported by pgBackRest. Remove all logic associated with 9.4 and update the tests.
This includes a small fix in infoPg.c to allow backup.info files with old versions to be saved. This allows expire to function when old versions are present. Even though those older versions cannot be used, they can be expired.
Tests for 9.4 are left in the expire/info tests to demonstrate that these commands work with old versions present.
NOTE TO PACKAGERS: This is last feature release to support the autoconf/make build. Please migrate to meson if you have not already done so. 2.54.X patch releases (if any) will continue to support autoconf/make.
Bug Fixes:
* Fix PostgreSQL query performance for large datasets. (Fixed by Thibault Vincent, David Steele. Reviewed by David Christensen, Antoine Millet. Reported by Antoine Millet.)
Features:
* Allow repositories on versioned storage to be read at a target time. (Reviewed by Stefan Fercot, David Christensen.)
* Allow requested standby backup to proceed with no standby. (Reviewed by Stefan Fercot.)
Improvements:
* Summarize backup reference list for info command text output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Refresh web-id token for each S3 authentication. (Contributed by Brent Graveland. Reviewed by David Steele.)
* Correctly display current values for indexed options in help. (Reviewed by David Christensen.)
* Save backup.info only when contents have changed. (Reviewed by Stefan Fercot.)
* Remove limitation on reading files in parallel during restore. (Reviewed by David Christensen.)
* Improve SFTP error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
Documentation Features:
* Add performance tuning section to user guide. (Reviewed by Stefan Fercot.)
Documentation Improvements:
* Clarify source for data_directory. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Matthias.)
* Better logic for deciding when a summary should be lower-cased. (Suggested by Daniel Westermann.)
There have been occasional SFTP authentication failures on 32-bit. We are planning to drop 32-bit support so it does not seem worth chasing these errors down and they are likely timing issues anyway.
In the case of a rapid restart it is possible that the socket may not be immediately available, so retry until it becomes available.
This is particularly useful for testing where sockets are bound and released very rapidly.
Option help summaries do not have initial capitals (except in special cases) and final periods so it makes sense to render the command summaries the same way.
Use the same function for both so they are consistent.
The asynchronous logic used to implement the query timeout was misusing PQisBusy(), which caused the wait handler to throttle the consumption of command results. It could introduce a large delay on a query up to `db-timeout` because of the back-off sequence.
Following the recommendation of libpq, fix by polling the client socket for data availability and then continue consuming results and checking for command busyness.
This is useful for code that has its own wait mechanism, e.g. poll(), but still needs a way to track overall time elapsed.
To keep it simple waitRemains() is called by waitMore().
Ubuntu 20.04 has been having consistent errors starting PostgreSQL 10 so move 9.5 to this container instead. An older version makes sense with an older distro.
Also move PostgreSQL 12 from RHEL 8 since this version will be EOL soon.
The current value for an indexed option was always for the first index, e.g. pg1-path. This is likely legacy from before indexing was added (and faithfully copied over from Perl, apparently).
Fix this by enumerating the current values in the option help and displaying <multi> in the option list when more than one value exists.
Options that are only valid on the command-line should not appear in the configuration reference because it implies that they can be added to pgbackrest.conf, which is not the case.
Most command-line options were already excluded because they lacked a section, but a few were slipping through.
This feature allows the archive-get, info, repo-get, repo-ls, restore, and verify commands to operate at a point-in-time on versioned buckets in Azure, GCS, and S3. This allows recovery even if a repository has been accidentally or maliciously deleted or corrupted.
Add a "prefer" value to the backup-standby option to allow the backup to proceed when no standby is found. Note that this will not help if the standby is responding but fails to sync with the primary after the backup has started.
Introduce a new option modifier, bool-like, that allows a boolean option to be converted to a string or string-id option while still allowing the option to act like a boolean on the command-line, e.g. --no-backup-standby.
This prevents backup.info from being saved again when expire does not make any changes.
More importantly, as we look to support versioning on object stores, it will be much easier to determine a good point-in-time to use for restore if there are no extraneous saves of backup.info.
The token file pointed to by the AWS_WEB_IDENTITY_TOKEN_FILE environment variable was read once at startup, but for long operations the token might expire before completion.
Instead read the token on each S3 authentication so the current token is always used.
Move tests that use functions that are not valid for non-Posix repositories to pg storage.
This allows for a test storage driver that only implements functions required for repositories.
Add getters where needed and remove the unused driver member from the StorageRead/Write objects. The new getters are only required for testing but they don't compromise the core code.
The rm would miss files beginning with a dot which are being used going forward for the versioning test driver.
Instead use find since it will delete all files.
This function will be needed elsewhere going forward and in any case it makes sense to move this out of the restore module.
Refactor to avoid dependency on a regular expression.
When restore was run as the root user the pg_control file would end up with root permissions. This bug was introduced in e634fd8. Fix this by directly overwriting the pg_control temp file rather than doing an atomic write that updates permissions. Also update other parameters to more closely match similar calls.
There was also an adjacent error where restore as the root user would fail if the base path did not exist. Fix this by ignoring the missing path since it will be created later and this logic is just trying to find an alternate user for permissions if the user in the manifest does not exist.
The backup reference list can be very long so it seems better to summarize the list by default for text output and keep the full list when --set is specified.
The none command was a bit confusing since it was only valid when parsing failed but still needed to be added to various switches and logic. Replace with cfgInited() which should make it clearer what state configuration is in.
Make the default command help and convert --version and --help to real options.
Combine version and help output into a single function to simplify processing in main.
Additional reformatting and a bit of refactoring.
This refactor simplifies the main() functions and puts the more commonly run commands first.
For core main() also remove code duplication in local/remote role handling.
Previously requesting help for roles other than main would result in a segfault since help.auto.c.inc does not include such help.
Prevent this by erroring when a user requests help for roles other than main.
This is a bug but it is unlikely to have ever been seen in the field and in any case it has a very low severity since it only impacts a corner-case for the help command.
This command was used by the Perl integration tests to create buckets for storage types that required it. Now that the integration tests are written in C they can simply use the same code to create buckets.
The command was also used in the documentation but there it seems more appropriate to use the corresponding vendor CLI.
Previously it was not possible to read or write two files at the same time on the same remote because the protocol was entirely taken over by the read or write command. Multiple reads are required to make restores efficient when a list of bundled files is being read but blocks need to be retrieved from a separate file or a different part of the same file.
Improve that situation with sessions that allow related commands to be run with shared state. Also break read/write into separate requests (rather than pushing all data at once) so they can be multiplexed.
The disadvantage for read/write is that they now require more back and forth to transfer a file. This is mitigated by sending asynchronous read/write requests to keep both server and client as busy as possible. Reads that can fit into a single buffer are optimized to transfer in a single command. Reads that transfer the entire file can also skip the close command since it is implicit on end-of-file.
These changes allow the protocol to be simplified to provide one response per request, which makes the data end message obsolete. Any data sent for the request is now added to the parameters so no data needs to be sent separately to the server outside the request parameters.
Also update the Db protocol to use the new sessions. Previously this code had tracked its own sessions.
IMPORTANT NOTE: The log-level-stderr option default has been changed from warn to off. This makes it easier to capture errors when only redirecting stdout. To preserve the prior behavior set log-level-stderr=warn.
NOTE TO PACKAGERS: The lz4 library is now required by the meson build.
NOTE TO PACKAGERS: Compiler support for __builtin_clzl() and __builtin_bswap64() is now required by the meson build.
Bug Fixes:
* Fix SFTP renaming failure when file already exists. (Fixed by Reid Thompson. Reviewed by David Steele. Reported by ahmed112212.)
Features:
* Allow backups to run concurrently on different repositories. (Reviewed by Reid Thompson, Stefan Fercot.)
* Support IP-based SANs for TLS certificate validation. (Contributed by David Christensen. Reviewed by David Steele.)
Improvements:
* Default log-level-stderr option to off. (Reviewed by Greg Sabino Mullane, Stefan Fercot.)
* Allow alternative WAL segment sizes for PostgreSQL ≤ 10. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
* Add hint to check SFTP authorization log. (Contributed by Vitalii Zurian. Reviewed by Reid Thompson, David Steele.)
Documentation Improvements:
* Clarify archive-push multi-repo behavior. (Reviewed by Stefan Fercot.)
Since 1141dc20 it has been possible to request that cfgParse() skip loading the config file. Use this logic to replace the code used to ignore config files in doc/test config load.
