1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4436 Commits

Author SHA1 Message Date
David Steele
49e252f492 Begin v2.53 development. 2024-05-27 11:49:21 +10:00
David Steele
dfc14b1934 v2.52: PostgreSQL 17 support
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.

Features:

* Add GCS batch delete support. (Reviewed by Reid Thompson.)
* S3 SSE-C encryption support. (Reviewed by Tim Jones. Suggested by Tim Jones.)
* PostgreSQL 17 support. (Reviewed by Stefan Fercot.)

Improvements:

* Allow explicit disabling of optional dependencies in meson builds. (Contributed by Michael Schout. Reviewed by David Steele.)
* Dynamically find python in meson build. (Contributed by Michael Schout. Reviewed by David Steele.)
* Tag pgbackrest build target in meson as installable. (Contributed by Bradford Boyle. Reviewed by David Steele.)

Documentation Improvements:

* Update start/stop documentation to reflect actual functionality. (Reviewed by Stefan Fercot.)
2024-05-27 11:13:16 +10:00
David Steele
899b892788 New CI container build for PostgreSQL 17 beta1.
Update the catalog version for beta 1 so pgbackrest will not work with any prior development versions.

Also improve the integration/all test so the catalog version does not need to be updated again during the beta period.
2024-05-24 12:24:11 +10:00
David Steele
c401ae8fb8 Explicitly run test servers on ipv4 localhost.
If a host defaults to ipv6 then it can confuse the tests and lead to connection failures and inconsistent error messages.

For now just hard-code the servers to run on ipv4 but this is an area for later improvement.
2024-05-23 11:03:59 +10:00
David Steele
04b0437976 Exclude documentation code coverage from metrics report.
Coverage of the documentation code is not important enough to report to users. If it were reported it should be in a separate section (along with test code coverage).
2024-05-23 10:58:52 +10:00
David Steele
91156bf7e1 Fix comment typos and formatting. 2024-05-23 08:19:50 +10:00
David Steele
b6ac11beec
Update start/stop documentation to reflect actual functionality.
The exact functionality of start/stop has evolved over time and has become a bit confusing. It may be appropriate to make the behavior more consistent but for now at least document the behavior correctly. The documentation for start/stop was fairly inaccurate.
2024-05-23 08:17:02 +10:00
David Steele
9910a4eddb Add hrnTzSet() to set timezone in tests.
3c8819e1 replaced gmtime/localtime with gmtime_r/localtime_r but did not take into account a subtle difference in how they operate. While gmtime/localtime operate as if tzset() has been called, i.e. they operate on the TZ env variable directly, gmtime_r/localtime_r require tzset() to be called after changing TZ for consistent results.

Rather than call tzset() every time TZ is changed, add hrnTzSet() to encapsulate both operations.
2024-05-20 08:44:37 +10:00
David Steele
28ad1badd9 Update Fedora test image to Fedora 40. 2024-05-18 18:55:53 +10:00
David Steele
ffe9a17fcb Replace erroneous THROW_SYS_ERROR_FMT() in storageInfo().
This was copied from storagePosixInfo() in a474ba54 but there is no guarantee that errno will be valid at this point. In most cases errno was zero so no system error message was displayed, but when using the Posix driver it could output "[2] No such file or directory". For other drivers errno was generally not set but could output a random error message in that case that errno was set by some unrelated action.

Use THROW_FMT() instead since errno will not always be set correctly and in any case "[2] No such file or directory" is not very useful information since the main error message already says that.

While this is technically a bug it is so harmless that it doesn't merit mention in the release notes.

This was discovered while testing on Fedora 40 which threw "[38] Function not implemented" -- clearly unrelated to missing paths/files.
2024-05-18 14:33:40 +10:00
David Steele
9e477c4321 Update GitHub test checkout and codeql action versions.
checkout@v3, codeql-action/init@v2, and codeql-action/analyze@v3 are now deprecated.
2024-05-18 09:59:32 +10:00
David Steele
db4eefae1a Remove ASSERT() in sckClientOpen() to silence Coverity complaint.
Coverity complains that this comparison might have a side effect because the variable is volatile. It's hard to see what that might be but since the assertion is not all that important, just remove it. During testing this sort of error will generally be caught by valgrind.
2024-05-10 10:01:16 +09:30
David Steele
d32eb5bb54 Silence invalid Coverity complaint in jsonReadPush().
Coverity complains that the output from THROW_FMT will be unpredictable since the order of operations in the call is not deterministic, but it fails to understand that subsequent calls to jsonReadTypeNextIgnoreComma() are noops until the value has been processed.

Silence Coverity by assigning the actual type to a local variable so jsonReadTypeNextIgnoreComma() is only called once.

Also fix an adjacent comment typo.
2024-05-10 09:45:01 +09:30
David Steele
bb8988b551 Update storage (except remote) module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on.

The storage/remote module will be updated with the protocol module once a major waiting refactor has been committed.
2024-04-27 19:14:01 +10:00
David Steele
76bcb740b6
Add GCS batch delete support.
The GCS driver sent a single file delete request for each file while deleting a path. Depending on latency this could lead to rather long delete times, especially noticeable during expiration.

Improve GCS delete to use batches, which require multipart HTTP, so also add multipart HTTP infrastructure.
2024-04-27 15:42:10 +10:00
Michael Schout
e00e33b528
Dynamically find python in meson build.
This is better than requiring a python3 binary to be on the path because some installations might have, e.g. python3.9.

Also add the python3-distutils package to Debian builds to make this work.
2024-04-25 19:32:07 +10:00
David Steele
55e996912a Update meson minimum version to 0.47.
This allows enabling the check option for run_command, which automatically fails when the command fails.
2024-04-25 18:35:48 +10:00
Reid Thompson
4ea4e3f380
Update Fedora test image to Fedora 38.
This should have been done in 434938e3 but somehow it didn't happen.

Fedora 38 requires 2048 bit keys so update the VM builds to use them. Update the documentation to use 2048 bit keys. This is not technically required by this commit but it makes sense to do it now.

Also update the key location for the yum.p.o repository.

Lastly, shuffle test PostgreSQL versions since PostgreSQL 11 is not longer available in the yum.p.o repository.
2024-04-23 22:34:32 +10:00
David Steele
b40c2616c2 Update info module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on.
2024-04-21 13:16:53 +10:00
David Steele
c6fcc81db6 Update db/postgres modules to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 13:01:40 +10:00
David Steele
19411f39d2 Update config module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 12:06:39 +10:00
David Steele
0e36e1a2fc Update common module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 12:01:01 +10:00
David Steele
987e6e31f4 Move packager note about meson to new version.
Since there were some issues found with the meson install (7877983a, 7b95fd3b) it makes sense for any packagers who have not made the migration to hold off until the next release.

Move the note to the next release where hopefully all issues have been addressed.
2024-04-21 09:42:11 +10:00
David Steele
9546b9c5d0 Update command module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-20 14:19:27 +10:00
David Steele
fb22f04555
PostgreSQL 17 Support.
Add catalog version and WAL magic for PostgreSQL 17.
2024-04-18 10:56:24 +10:00
David Steele
c8cf8e1f2b Update contributor name. 2024-04-17 19:02:28 +10:00
udf2457
57731b6cd2
S3 SSE-C encryption support.
This feature (enabled with --repo-s3-sse-customer-key) provides an encryption key to encrypt the data after it has been transmitted to the server.

While not as secure as encrypting data before transmission (--repo-cipher-type), this may be useful in certain configurations.
2024-04-17 11:58:13 +10:00
David Steele
06d3cb767c Improve unit testing of valid page sizes in pg_control.
Spread the tests over more versions of PostgreSQL and improve/fix comments.
2024-04-17 11:43:41 +10:00
Michael Schout
7b95fd3bd2
Allow explicit disabling of optional dependencies in meson builds.
On some platforms, e.g. FreeBSD, there is a requirement to allow the user to disable support for features even when the required library is present.

Introduce tri-state options for the optional features: auto mimics the current behavior and is the default, enable requires libraries for the feature to be present, and disable disables the feature without checking the libraries.
2024-04-10 09:23:17 +10:00
David Steele
dab52739cd Fix comment typo. 2024-04-04 18:21:26 +11:00
David Steele
fec1b215e3 Remove test data files made obsolete by d41b21c8.
These should have been removed when the mock integration tests were removed.

Ideally we would also remove filecopy.table.bin but it serves to provide realistic page data for performance testing.
2024-03-30 10:43:05 +11:00
David Steele
9f5a97139f Allow strIdToLog() to output "null" values.
A valid StringId can never be zero so it more or less serves as a NULL value. In most cases zero will not be valid, but it is better to catch this condition with an assert rather than an error in logging.
2024-03-29 12:21:06 +11:00
David Steele
014e24889c Remove extra space before colons in meson.build files.
The spacing was not consistent so use the style that best matches our general coding standards.
2024-03-27 09:53:49 +11:00
Bradford Boyle
7877983acb
Tag pgbackrest build target in meson as installable.
By default meson does not install anything. Targets can be installed by tagging them as installable in the build definition.
2024-03-27 09:29:37 +11:00
David Steele
924aa5e8b1 Begin v2.52 development. 2024-03-25 11:22:53 +13:00
David Steele
dc07fb1e5b v2.51: Meson Build System
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.

Bug Fixes:

* Skip zero-length files for block incremental delta restore. (Reviewed by Sebastian Krause, René Højbjerg Larsen. Reported by Sebastian Krause.)
* Fix performance regression in storage list. (Reviewed by Stephen Frost. Reported by Maksym Boguk.)
* Fix progress logging when file size changes during backup. (Reviewed by Stephen Frost. Reported by samkingno.)

Improvements:

* Improved support for dual stack connections. (Reviewed by Stephen Frost. Suggested by Timothée Peignier.)
* Make meson the primary build system. (Reviewed by Stephen Frost.)
* Detect files that have not changed during non-delta incremental backup. (Reviewed by Stephen Frost.)
* Prevent invalid recovery when backup_label removed. (Reviewed by Stephen Frost.)
* Improve archive-push WAL segment queue handling. (Reviewed by Stephen Frost.)
* Limit resume functionality to full backups. (Reviewed by Stephen Frost, Stefan Fercot.)
* Update resume functionality for block incremental. (Reviewed by Stephen Frost.)
* Allow --version and --help for version and help. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add detailed backtrace to autoconf/make build. (Reviewed by Stephen Frost.)

Documentation Improvements:

* Update references to recovery.conf. (Reviewed by Stefan Fercot. Suggested by Stephen Frost.)
2024-03-25 09:53:22 +13:00
David Steele
76956e71cf
Fix progress logging when file size changes during backup.
If the file size changed during backup then the progress percentage in the log would not be accurate.

Fix this by using the original size to increment the progress since progress total was calculated from original file sizes.
2024-03-20 09:05:39 +13:00
David Steele
de55902fb3 Redact secure options in check --report.
For this to be practically useful secure options must be redacted. Otherwise, no user is likely to share the report.

Since this feature is still internal, there is no real world impact.
2024-03-13 12:06:23 +13:00
David Steele
ff47450402
Update resume functionality for block incremental.
Resume was not updated for block incremental so block incremental files were always removed during a resume. Resume worked but was very inefficient with block incremental enabled.

Update resume to preserve block incremental files and add tests.
2024-03-12 12:41:58 +13:00
David Steele
e634fd85ce
Prevent invalid recovery when backup_label removed.
If backup_label is removed from a restored backup then PostgreSQL will instead use checkpoint information from pg_control to attempt (what is thinks is) crash recovery. This will nearly always result in a corrupt cluster because the checkpoint will not be from the beginning of the backup, and even if it is, the end point will not be specified, which could lead to recovery stopping too early.

To prevent this, invalidate the checkpoint LSN in pg_control on restore. If backup_label is removed then recovery will still fail because PostgreSQL will not be able to find the invalid checkpoint. The LSN of the checkpoint is not logged but it will be visible in pg_controldata output as 0/DEAD. This value is invalid because PostgreSQL always skips the first WAL segment when initializing a cluster.
2024-03-10 17:08:42 +13:00
David Steele
960b43589d Add validation for WAL segment size in pg_control.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being near the end and containing a limited number of discrete values.
2024-03-10 16:17:50 +13:00
David Steele
63541b2273 Add validation for page checksum version in pg_control.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being all the way at the end and being four bytes that can only have one of two values. Something more distinctive than 0 and 1 would be better, but this is what we have to work with.

Convert PgControl.pageChecksum to unsigned int and rename to PgControl.pageChecksumVersion and make all downstream changes required for the new datatype.
2024-03-10 15:50:10 +13:00
David Steele
7448fde157
Improved support for dual stack connections.
Connections are established using the "happy eyeballs" approach from RFC 8305, i.e. new addresses (if available) are tried if the prior address has already had a reasonable time to connect. This prevents waiting too long on a failed connection but does not try all the addresses at once. Prior connections that are still waiting are rechecked periodically if no subsequent connection is successful.

This improves substantially on 39bb8a0, which failed to take into account connection attempts that do not fail (but never connect) and use up all the available time.
2024-03-10 11:36:39 +13:00
David Steele
f287178b70
Set function/variable visibility to hidden.
This saves about 16KiB in the binary and reduces exported symbols by about 75%. All variables are still exported and any functions that are referenced by their pointers or extern'd but never used outside the module where they are defined.

In addition to modest space savings, this should also increase performance a bit since the compiler can simplify calls to these functions and load the binary should also be a little faster.

The GCC documentation does not make it clear that visibility can be used with variables, but it certainly makes a difference in the binary size, so something is happening. Other sources on the internet suggest that visibility can be used with variables. Clearly exports are not affected, but there may be some other optimization happening.
2024-03-10 11:09:13 +13:00
David Steele
e3d9df3ae9
Make meson the primary build system.
Meson has a lot of advantages over autoconf/make, primarily in ease-of-use and performance. Make meson the only build system used for testing and building the Debian documentation, but leave the RHEL documentation using autoconf/make for now so it gets some testing.
2024-03-10 10:53:31 +13:00
David Steele
c64cd8e019 Disable arm64 test in CirrusCI.
There seems to be a shortage of arm64 hosts because queue times have been steadily increasing over the last few weeks. It can now take several hours to get an arm64 test queued, which makes it difficult to get development done.

Disable for the time being and hope the resource issue gets resolved in the future.
2024-03-10 10:43:24 +13:00
David Steele
eda7706f53 Make valgrind libssh2 leak check less specific.
The leak kind is usually definite but sometimes flaps to possible. For stability purposes accept any leak kind.

Note that this is a leak in a specific version of libssh2 and not a bug in pgBackRest.
2024-03-10 09:40:13 +13:00
David Steele
dddcbcd8e9
Limit resume functionality to full backups.
Resume does not work correctly with delta diff/incr backups because the presence of a reference causes it to remove the file with the idea that it can just be referenced again. This is true for timestamp-based backups but for deltas all existing files need to be rechecked (which requires a reference).

This is fixable but not without significant effort and new tests and it calls into question the usefulness of non-full resumes. For diff/incr, if the file was changed since the prior backup there is a good chance it will be modified again before the resume occurs.

In order to keep this feature as useful as possible for the most valuable case, limit resumes to full backups.
2024-03-10 09:32:55 +13:00
David Steele
9d91d1b2f8
Detect files that have not changed during non-delta incremental backup.
02eea55 added code to load a buffer of data from a file being backup up to detect files that have been truncated to zero after manifest generation. This mechanism can also be used to detect files that have not changed since the prior backup.

If the result of the file copy fits into a single buffer, then the size and checksum can be compared to the prior file before anything gets stored. If the file matches then it is referenced to the file in to prior backup.

The size that can be compared for normal copies is limited by the buffer size but for block incremental it works with any size file since there is no output from block incremental when the file is identical.
2024-03-08 15:07:43 +13:00
David Steele
cf17515e40
Improve archive-push WAL segment queue handling.
Infer the size of all WAL segments from the size of the first segment rather than getting info for all segments (up to queue size). If the segments are not the same size then there are larger issues than the WAL queue.
2024-03-08 12:34:11 +13:00