PostgreSQL 13 is EOL and no longer available in the yum.postgresql.org repository.
Update the base and upgrade versions of the RHEL and Debian documentation to better cover supported versions.
The unit tests do not require this field to be non-zero so in general it has not been set.
However, in production this field is always set and can be used to calculate timestamp deltas, which will be useful in a future commit.
Commit the test changes separately to show that there is no change in behavior.
The prior code issued a seek for each block even if the file was in the correct position. The extra call to lseek() was probably not too expensive but a flush was also required in case of a seek on the next loop. Flushing the write buffer after each block (which is often 8KiB) was definitely wasteful.
Fix this by adding seek to the IoWrite interface. That means the file position can be tracked and calls to seek/flush are only done when needed.
Seek during write is only used in PGDATA during restore so seek functionality is only added to the Posix driver.
This check was lost in the migration from autoconf which means that the UNCONSTIFY() macros has not been enforcing since then.
Add the check so UNCONSTIFY() enforces as expected.
Putting these defines in meson.build is a bit annoying because of the added syntax and double commenting. Since the defines are static there is no good reason to have them there.
This also creates a place for test builds to add global defines -- again rather than in meson.build.
Both of these lists were sorted descending so all comparisons in the comparator had to be reversed. This made it hard to reason about sort order.
It makes more sense to sort ascending and have the comparisons also be ascending. The final sort order remains the same.
The restore module has been large and unwieldy for some time but I have been loathe to split it into separate compilation units because it means maintaining additional header files, updating build files, and losing optimizations from static functions. These functions are only used internally by restore and it seems wasteful to extern them.
We already have a number of cases where C files are included directly into other C files, especially .vendor.c.inc and .auto.c.inc files. We also include C files to add functionality needed for build/doc/test to core objects without having to add that functionality to core. See src/build/common/string.c for an example.
The test/coverage code already supports C includes but I had to update it to recognize the new extension.
Moving initialization to the interface consolidates code and removes duplication in the drivers. This will become critical as the interface manipulates multiple drivers to improve performance.
This refactor is less important for StorageWrite but it seems better to keep them consistent.
This made sense during the days when the project was juggling Perl code, a C binary, and a C library. These days the build is far simpler and this dependency just means build failures when salsa.debian.org is cranky (often) without any apparent benefit.
The temp mem context block was outside the if statement that determines if a file needs to be copied or not, which meant that some mem contexts would never be used.
Reverse this order to avoid creating unused mem contexts.
This was not a bug since the Posix/SFTP drivers do not allow read retries but it could easily turn into a bug if retries are enabled in the future so it seems safer to move the logic to a location that will work with retries.
Designated initializers were being used to initialize the structs but not the arrays. Adding designated initializers makes the code a little clearer.
In the packTypeMapData array designated initializers also allow removal of the explicit zero padding and make initialization order unimportant (though it is better to keep them in order to match the enum).
ed390780 added this warning but included an extra dash which caused it to be ignored. Remove the extra dash so the warning is applied.
Also remove some unused test code that violated the new warning.
Previously -Wextra was used, which is overly broad and led to the issue in 303e7458. Instead use a specific error to avoid side effects.
It is possible that this warning will not be recognized by all compilers but that is not an issue since this is only important for development. Warnings are ignored for production builds.
The main() parameters in meson compiler probes were not being used but could cause unused parameter errors.
Since the parameters are not needed remove them. This also has the benefit of making the code a bit simpler.
It is convenient that a StringId can be used as an integer and also be rendered as a string. However, in many cases it is simpler and more efficient to have a sequence that can be used for lookup into an array or in a switch statement. This is particularly useful for StringIds generated by the config module.
Sequences cannot be used everywhere, though. In some cases StringIds are generated from text stored in files (e.g. compress type) in which case it is not possible to add sequence info without additional complexity.
IMPORTANT NOTE: The minimum values for the repo-storage-upload-chunk-size option have increased. They now represent the minimum allowed by the vendors.
Bug Fixes:
* Fix deadlock due to logging in signal handler. (Fixed by Maxim Michkov. Reviewed by David Steele.)
Features:
* HTTP support for S3, GCS, and Azure. (Contributed by Will Morland. Reviewed by David Steele.)
* Allow expiration of oldest full backup regardless of current retention. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Ron Johnson.)
* Support for Azure managed identities. (Contributed by Moiz Ibrar, Matthew Mols. Reviewed by David Steele.)
* Experimental support for S3 EKS pod identity. (Contributed by Pierre BOUTELOUP. Reviewed by David Steele.)
* Allow configuration of TLS cipher suites. (Contributed by Gunnar "Nick" Bluth. Reviewed by David Steele.)
* Allow process priority to be set. (Reviewed by Douglas J Hunley.)
Improvements:
* Allow dots in S3 bucket names when using path-style URIs. (Contributed by Joakim Hindersson. Reviewed by David Steele.)
* Require TLS >= 1.2 unless verification is disabled. (Reviewed by Douglas J Hunley, Gunnar "Nick" Bluth.)
* Dynamically size S3/GCS/Azure chunks for large uploads. (Reviewed by Douglas J Hunley. Suggested by Timothée Peignier.)
* Optimize S3/GCS/Azure chunk size for small files. (Reviewed by Douglas J Hunley.)
* Remove support for PostgreSQL 9.5. (Reviewed by Douglas J Hunley.)
* Improve logging of default for options with an unresolved dependency. (Reviewed by Stefan Fercot.)
Documentation Improvements:
* Remove explicit max_wal_senders/wal_level configuration from user guide. (Suggested by Jamie Nguyen.)
* Clarify that bundling is useful for filesystems with large block sizes. (Suggested by Ron Johnson.)
The az tool used to create the bucket now requires an API version greater than what Azurite supports. This may be a mistake on their part (but has not resolved after a day) or may indicate that Azurite is no longer actively supported.
Either way, for now it is easiest to suppress the version check so CI builds can proceed. Other corrective action can be taken later as needed.
Now that the StringIds are not stored in parse.auto.c.inc it may not be obvious if an invalid StringId is generated. By moving the encoding to after the value has been validated we can error if the StringId is invalid rather than throwing an error that the value does not exist in the allow list.
Mostly this just makes the code cleaner but there are also a few function calls that were replaced with the macro.
Also tighten up the logic in cfgParseOptionValuePack() a bit even though is not related to this macro change.
These StringIds use space and increase churn when new ids are added. It is easy and efficient to use strings for comparison and convert to StringId in code.
Fetch credentials automatically using EKS pod identity, which removes the need for static configuration. Credentials are automatically updated before they expire to support long-running commands.
Since the server exits on error in this test there is no need to shut it down when the script completes. This usually worked because the message would arrive before the server had shutdown completely but sometimes it would error on a broken pipe.
Configuration of specific cipher suites may be required for compliance or to use preferred ciphers for security.
Cipher suites are applied to the entire process and cannot be configured on a per-connection basis, except that for object store clients (e.g. S3) verification can be disabled.
TLS 1.0 and 1.1 have been deprecated since 2021 (see RFC-8996) and TLS 1.2 has been available since 2008. As such it makes sense to require TLS >= 1.2 when verification is enabled. Verification is always enabled for TLS protocol sessions within pgBackRest but can be disabled for object stores (.e.g. S3) to support self-signed certificates on internal servers.
There is a slight change in behavior when verification is disabled. In prior versions SSL 2/3 would be disabled but now they are allowed (as well as TLS 1.0/1.2). With verification disabled it doesn't seem useful to be picky about protocol versions and disabling TLS 1.0/1.1 could easily cause breakage on older TLS servers.
Fetch credentials automatically using managed identities, which removes the need for static configuration. Credentials are automatically updated before they expire to support long-running commands.
In practice StringIds that could not be output as the original string were not useful. Remove this functionality to simplify the code and reclaim the bit for other purposes.
Functions for creating String and StringZ objects should be in their respective modules so move them there. This simplifies the dependencies for StringId and is more modular.
Moving these constants to stringStatic.h reduces the dependency on stringZ.h for low-level debug logging. This makes it possible to add new capabilities to stringZ.c.
After JsonRead was introduced the performance-sensitive areas of info were migrated from Variant. However, other areas were left using Variant because they were not important enough to update at that time.
Migrate remaining Variant usage to JsonRead wherever possible for consistency. Also improve memory management to avoid some cruft that would end up in the object mem context and avoid switching to the object mem context when possible.
David Steele
Yegor Zhmak
wolrajhti
Gunnar "Nick" Bluth
Moiz Ibrar