Typically we use the oldest Debian/Ubuntu to run 32-bit unit and integration tests. However, 32-bit is no longer fully supported by Ubuntu (multiple packages we need are missing) and apt.postgresql.org no longer packages for any 32-bit version.
To address these changes, do 64-bit integration testing on the oldest Debian/Ubuntu (currently Ubuntu 20.04) and 32-bit unit/integration testing on the oldest Debian (currently 11) using the included version for integration testing.
Mock integration tests were removed in d41b21c8 but CI was still trying to run them, so remove from CI.
Also rename mock to integration in test unit tests to avoid confusion in the future.
The prior SAN code only recognized DNS-based SANs, which meant that it would not properly validate if using an IP-based SAN.
Add support for IPv4 and IPv6 SANs with exact matching only.
This simplifies testing when certificate generation tools have trouble generating a DNS:1.2.3.4-style SAN, preferring to include the SAN as IP:1.2.3.4.
Reduce redundancy by improving cfgParseCommandId() to work when a command role is present. This way the function does not need to be called twice.
Also, remove the use of StringList in cfgParse() since checking for a colon is faster and saves memory allocations. Modify cfgParseCommandRoleEnum() to accept char * since a String is no longer produced for the role name.
Errors in code generation can cause the test binary build to fail and then it is not possible to see the generated code.
Instead, generate code first so any errors can be seen and analyzed.
Create mappings between integer, size, time, and stringid option values and their string equivalents. This allows for better error messages and means that the mappings do not need to be stored with defaults, allow lists, etc.
Update error handling for libssh2_sftp_rename_ex() in storageWriteSftpClose() when a file already exists.
The SFTP servers used during development and testing never returned LIBSSH2_FX_FILE_ALREADY_EXISTS, rather they returned LIBSSH2_FX_FAILURE when a file already existed. However, it is clear that some SFTP servers use LIBSSH2_FX_FILE_ALREADY_EXISTS so add support.
The prior locking only allowed one backup per stanza, which was required by PostgreSQL <= 9.5 and didn't present a problem when only one stanza could be created.
Now that multiple stanzas are allowed relax this restriction so that backups can run concurrently for PostgreSQL > 9.5. To do this, update the locking to be per stanza and repo rather than per stanza. Remotes are not aware of the repos that require locking so send an explicit list of files to be locked to the remote. Also remove the advisory lock for PostgreSQL > 9.5.
For info output the running backups are combined for progress output in order to avoid changing the JSON format. It definitely makes sense to have per repo progress as well but that will be left for a future commit.
Similar to size options in 038abaa7, time option defaults and allowed values were displayed in seconds, which could be confusing when the values were large.
The time options were not updated in 038abaa7 because it required removing the ability to do fractional seconds, e.g. 0.5 seconds. In theory this could cause breakage for users but it seems really unlikely. Fractional seconds are used in tests, however, so the tests have been changed to use milliseconds where required, e.g. 500ms.
This default was being mistakenly inherited from job-retry and would set job-retry-interval to 1 second for archive get and push. In practice this did not matter since archive get and push default to one retry and the first retry interval is always 0, but it still makes sense to fix it to correct the behavior when the value of job-retry is changed.
Rather than using the full enum just use the part of the enum that is unique. This makes the output a bit more readable by removing the repetitive elements. The prefix for each enum is built into its macro.
Writing warnings and errors to stderr by default leads to error messages being lost when the user does not correctly redirect stderr while generating logs for analysis. This happens so often that it seems worth changing the default to increase the quality of the logs we receive.
If the user has explicitly set log-level-stderr then there is no change in behavior.
These functions will be useful for optimizing varint-128 functions.
Require them in the meson build before adding new code in case there are problems with packaging.
We would like to use lz4 for protocol compression instead of gz but first we need to make sure this is not going to cause a problem for packaging.
To do this make lz4 required in meson but make no changes to the code so this is an easy revert for packagers if there is an issue.
These were broken while code was being migrated to C and went unnoticed because the options are generally only used when doing performance testing.
The C code can only take one --run param so add a check for that in test.pl.
31c7824a should have added remote locks when the commands were modified to run remotely. This is unlikely to have caused issues since these commands are generally not run concurrently with backup/expire but having the locks is safer.
Refactor the lock module to split command-specific logic from the basic file locking functionality. Command specific logic is now in command/lock.c. This will make it easier to implement new features such as repository locking and updating lock file contents on remotes.
This implementation is essentially a drop-in replacement but there are a few differences. First, the lock names no longer require a path (the path is added in the lock module). Second, the timeout functionality has been removed since it was not being used.
If a file on the primary was larger than on the replica then the next diff/incr backup would store the primary size instead of the replica size when block incremental was enabled. On the next diff/incr backup this would lead to a repo size must be > 0 for file error when validating the manifest.
Fix this by limiting copy based on sizeOriginal rather than size so size can be set to the value expected to be stored in the manifest. As a bonus sizePrior is no longer needed since size can be used for the same purpose.
This means valgrind is no longer built from source, which caused image builds to run for a very long time.
Valgrind is only required in a few images for testing.
Alternative WAL segment sizes can be configured in PostgreSQL <= 10 with compile-time options. We have not allowed these before since it was not a well-tested feature of PostgreSQL.
However, forks such as Greenplum allow alternative WAL segment sizes at initdb time (which are presumably well-tested) so it makes sense to allow it.
Since the PostgreSQL versions in question are all EOL it is not important to have this restriction in place anymore.
These tests are important for an upcoming bug fix related to differing sizes of a file on a primary vs standby.
The test that demonstrates the bug cannot be included here since it causes a test failure, but this commit introduces the infrastructure and one test to guard against a regression in the bug fix.
Move the file.copy check before the manifest lookup to save a lookup and (probably) find for zero-length files when bundling.
This also removes a layer of indentation which helps with readability.
This simpler implementation only requires that a lock file begin with the stanza (followed by a dash) and end in .lock.
This will make the implementation more resilient to planned changes in lock file naming.
Since lockRead() was being called twice the state of the lock file could change between leading to invalid data in the info output.
Instead call lockRead() once and use the result for both the validity test and output.
lcov does not seem to be very well maintained and is often not compatible with the version of gcc it ships with until a few months after a new distro is released. In any case, lcov is that not useful for us because it generates reports on all coverage while we are mainly interested in missing coverage during development.
Instead use the JSON output generated by gcov to generate our minimal coverage report and metrics for the documentation.
There are some slight differences in the metrics. The difference in the common module was due to a bug in the old code -- build/common was being added into common as well as being reported separately. The source of the two additional branches in the backup module is unknown but almost certainly down to how exclusions are processed with regular expressions. Since there is additional coverage rather than coverage missing this seems fine.
Since this was pretty much a rewrite it was also a good time to migrate to C.
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.
Features:
* Add GCS batch delete support. (Reviewed by Reid Thompson.)
* S3 SSE-C encryption support. (Reviewed by Tim Jones. Suggested by Tim Jones.)
* PostgreSQL 17 support. (Reviewed by Stefan Fercot.)
Improvements:
* Allow explicit disabling of optional dependencies in meson builds. (Contributed by Michael Schout. Reviewed by David Steele.)
* Dynamically find python in meson build. (Contributed by Michael Schout. Reviewed by David Steele.)
* Tag pgbackrest build target in meson as installable. (Contributed by Bradford Boyle. Reviewed by David Steele.)
Documentation Improvements:
* Update start/stop documentation to reflect actual functionality. (Reviewed by Stefan Fercot.)
Update the catalog version for beta 1 so pgbackrest will not work with any prior development versions.
Also improve the integration/all test so the catalog version does not need to be updated again during the beta period.
If a host defaults to ipv6 then it can confuse the tests and lead to connection failures and inconsistent error messages.
For now just hard-code the servers to run on ipv4 but this is an area for later improvement.
Coverage of the documentation code is not important enough to report to users. If it were reported it should be in a separate section (along with test code coverage).
The exact functionality of start/stop has evolved over time and has become a bit confusing. It may be appropriate to make the behavior more consistent but for now at least document the behavior correctly. The documentation for start/stop was fairly inaccurate.
3c8819e1 replaced gmtime/localtime with gmtime_r/localtime_r but did not take into account a subtle difference in how they operate. While gmtime/localtime operate as if tzset() has been called, i.e. they operate on the TZ env variable directly, gmtime_r/localtime_r require tzset() to be called after changing TZ for consistent results.
Rather than call tzset() every time TZ is changed, add hrnTzSet() to encapsulate both operations.
This was copied from storagePosixInfo() in a474ba54 but there is no guarantee that errno will be valid at this point. In most cases errno was zero so no system error message was displayed, but when using the Posix driver it could output "[2] No such file or directory". For other drivers errno was generally not set but could output a random error message in that case that errno was set by some unrelated action.
Use THROW_FMT() instead since errno will not always be set correctly and in any case "[2] No such file or directory" is not very useful information since the main error message already says that.
While this is technically a bug it is so harmless that it doesn't merit mention in the release notes.
This was discovered while testing on Fedora 40 which threw "[38] Function not implemented" -- clearly unrelated to missing paths/files.
