1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4453 Commits

Author SHA1 Message Date
David Steele
6495a46ca3 Fix --run and --scale options for unit tests.
These were broken while code was being migrated to C and went unnoticed because the options are generally only used when doing performance testing.

The C code can only take one --run param so add a check for that in test.pl.
2024-06-25 15:46:07 +08:00
David Steele
cfb8aa202e Add remote locks for stanza commands missed in 31c7824a.
31c7824a should have added remote locks when the commands were modified to run remotely. This is unlikely to have caused issues since these commands are generally not run concurrently with backup/expire but having the locks is safer.
2024-06-21 13:29:17 +08:00
David Steele
b9a9ef2d5f Clarify when code generation is run when modifying config.yaml.
The contributing guide indicated that this happened at compile time but in fact it happens when test.pl is run.
2024-06-21 11:17:42 +08:00
David Steele
bcda7f8c7e Optimize assignment of command/option rules.
Assign the rule to a local variable to make subsequent assignments more efficient.

Also fix a missing const.
2024-06-20 13:06:04 +08:00
David Steele
a0a5f2300c Replace strftime() with cvtTimeToZP() and strNew/CatTimeP().
These functions produce cleaner code and hide implementation details.
2024-06-19 10:33:46 +08:00
David Steele
cec486b6dd Refactor backup to use flag for backup standby state rather than option.
Using the option adds a bit of complexity -- it is simpler just to use the state of the standby object to determine if backup from standby is enabled.
2024-06-18 11:42:59 +08:00
David Steele
270dce41b6
Refactor lock module.
Refactor the lock module to split command-specific logic from the basic file locking functionality. Command specific logic is now in command/lock.c. This will make it easier to implement new features such as repository locking and updating lock file contents on remotes.

This implementation is essentially a drop-in replacement but there are a few differences. First, the lock names no longer require a path (the path is added in the lock module). Second, the timeout functionality has been removed since it was not being used.
2024-06-18 10:43:54 +08:00
David Steele
ad7377c75b
Fix issue with files larger on the replica than on the primary.
If a file on the primary was larger than on the replica then the next diff/incr backup would store the primary size instead of the replica size when block incremental was enabled. On the next diff/incr backup this would lead to a repo size must be > 0 for file error when validating the manifest.

Fix this by limiting copy based on sizeOriginal rather than size so size can be set to the value expected to be stored in the manifest. As a bonus sizePrior is no longer needed since size can be used for the same purpose.
2024-06-16 11:55:04 +08:00
David Steele
2ec99ca4d9 Specify test images that get valgrind installed.
This means valgrind is no longer built from source, which caused image builds to run for a very long time.

Valgrind is only required in a few images for testing.
2024-06-12 16:08:17 +10:00
David Steele
886bb281f6 Migrate CentOS 7 integration tests to Rocky 8.
CentOS 7 will be EOL on June 30 and since there is no CentOS 8 migrate instead to Rocky 8.
2024-06-11 15:20:04 +10:00
Viktor Kurilko
4ac3b82c99 Allow alternative WAL segment sizes for PostgreSQL <= 10.
Alternative WAL segment sizes can be configured in PostgreSQL <= 10 with compile-time options. We have not allowed these before since it was not a well-tested feature of PostgreSQL.

However, forks such as Greenplum allow alternative WAL segment sizes at initdb time (which are presumably well-tested) so it makes sense to allow it.

Since the PostgreSQL versions in question are all EOL it is not important to have this restriction in place anymore.
2024-06-11 12:08:52 +10:00
David Steele
e8b965756c Add unit tests for backup from standby with block incremental.
These tests are important for an upcoming bug fix related to differing sizes of a file on a primary vs standby.

The test that demonstrates the bug cannot be included here since it causes a test failure, but this commit introduces the infrastructure and one test to guard against a regression in the bug fix.
2024-06-09 11:39:58 +10:00
David Steele
cf478bc753 Improve efficiency of incremental manifest build.
Move the file.copy check before the manifest lookup to save a lookup and (probably) find for zero-length files when bundling.

This also removes a layer of indentation which helps with readability.
2024-06-08 13:00:53 +10:00
David Steele
48823b6cd3 Simplify lock file scanning in stop command.
This simpler implementation only requires that a lock file begin with the stanza (followed by a dash) and end in .lock.

This will make the implementation more resilient to planned changes in lock file naming.
2024-06-07 14:24:08 +10:00
David Steele
ea1596152c Avoid possible race condition while reading lock files in info command.
Since lockRead() was being called twice the state of the lock file could change between leading to invalid data in the info output.

Instead call lockRead() once and use the result for both the validity test and output.
2024-06-07 14:09:17 +10:00
David Steele
01838995ce Replace tabs with spaces in meson.build files.
These were probably copied over from Makefile.in.
2024-06-07 09:21:13 +10:00
David Steele
6f562fba60 Migrate coverage testing to C and remove dependency on lcov.
lcov does not seem to be very well maintained and is often not compatible with the version of gcc it ships with until a few months after a new distro is released. In any case, lcov is that not useful for us because it generates reports on all coverage while we are mainly interested in missing coverage during development.

Instead use the JSON output generated by gcov to generate our minimal coverage report and metrics for the documentation.

There are some slight differences in the metrics. The difference in the common module was due to a bug in the old code -- build/common was being added into common as well as being reported separately. The source of the two additional branches in the backup module is unknown but almost certainly down to how exclusions are processed with regular expressions. Since there is additional coverage rather than coverage missing this seems fine.

Since this was pretty much a rewrite it was also a good time to migrate to C.
2024-05-31 14:52:07 +10:00
David Steele
49e252f492 Begin v2.53 development. 2024-05-27 11:49:21 +10:00
David Steele
dfc14b1934 v2.52: PostgreSQL 17 support
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.

Features:

* Add GCS batch delete support. (Reviewed by Reid Thompson.)
* S3 SSE-C encryption support. (Reviewed by Tim Jones. Suggested by Tim Jones.)
* PostgreSQL 17 support. (Reviewed by Stefan Fercot.)

Improvements:

* Allow explicit disabling of optional dependencies in meson builds. (Contributed by Michael Schout. Reviewed by David Steele.)
* Dynamically find python in meson build. (Contributed by Michael Schout. Reviewed by David Steele.)
* Tag pgbackrest build target in meson as installable. (Contributed by Bradford Boyle. Reviewed by David Steele.)

Documentation Improvements:

* Update start/stop documentation to reflect actual functionality. (Reviewed by Stefan Fercot.)
2024-05-27 11:13:16 +10:00
David Steele
899b892788 New CI container build for PostgreSQL 17 beta1.
Update the catalog version for beta 1 so pgbackrest will not work with any prior development versions.

Also improve the integration/all test so the catalog version does not need to be updated again during the beta period.
2024-05-24 12:24:11 +10:00
David Steele
c401ae8fb8 Explicitly run test servers on ipv4 localhost.
If a host defaults to ipv6 then it can confuse the tests and lead to connection failures and inconsistent error messages.

For now just hard-code the servers to run on ipv4 but this is an area for later improvement.
2024-05-23 11:03:59 +10:00
David Steele
04b0437976 Exclude documentation code coverage from metrics report.
Coverage of the documentation code is not important enough to report to users. If it were reported it should be in a separate section (along with test code coverage).
2024-05-23 10:58:52 +10:00
David Steele
91156bf7e1 Fix comment typos and formatting. 2024-05-23 08:19:50 +10:00
David Steele
b6ac11beec
Update start/stop documentation to reflect actual functionality.
The exact functionality of start/stop has evolved over time and has become a bit confusing. It may be appropriate to make the behavior more consistent but for now at least document the behavior correctly. The documentation for start/stop was fairly inaccurate.
2024-05-23 08:17:02 +10:00
David Steele
9910a4eddb Add hrnTzSet() to set timezone in tests.
3c8819e1 replaced gmtime/localtime with gmtime_r/localtime_r but did not take into account a subtle difference in how they operate. While gmtime/localtime operate as if tzset() has been called, i.e. they operate on the TZ env variable directly, gmtime_r/localtime_r require tzset() to be called after changing TZ for consistent results.

Rather than call tzset() every time TZ is changed, add hrnTzSet() to encapsulate both operations.
2024-05-20 08:44:37 +10:00
David Steele
28ad1badd9 Update Fedora test image to Fedora 40. 2024-05-18 18:55:53 +10:00
David Steele
ffe9a17fcb Replace erroneous THROW_SYS_ERROR_FMT() in storageInfo().
This was copied from storagePosixInfo() in a474ba54 but there is no guarantee that errno will be valid at this point. In most cases errno was zero so no system error message was displayed, but when using the Posix driver it could output "[2] No such file or directory". For other drivers errno was generally not set but could output a random error message in that case that errno was set by some unrelated action.

Use THROW_FMT() instead since errno will not always be set correctly and in any case "[2] No such file or directory" is not very useful information since the main error message already says that.

While this is technically a bug it is so harmless that it doesn't merit mention in the release notes.

This was discovered while testing on Fedora 40 which threw "[38] Function not implemented" -- clearly unrelated to missing paths/files.
2024-05-18 14:33:40 +10:00
David Steele
9e477c4321 Update GitHub test checkout and codeql action versions.
checkout@v3, codeql-action/init@v2, and codeql-action/analyze@v3 are now deprecated.
2024-05-18 09:59:32 +10:00
David Steele
db4eefae1a Remove ASSERT() in sckClientOpen() to silence Coverity complaint.
Coverity complains that this comparison might have a side effect because the variable is volatile. It's hard to see what that might be but since the assertion is not all that important, just remove it. During testing this sort of error will generally be caught by valgrind.
2024-05-10 10:01:16 +09:30
David Steele
d32eb5bb54 Silence invalid Coverity complaint in jsonReadPush().
Coverity complains that the output from THROW_FMT will be unpredictable since the order of operations in the call is not deterministic, but it fails to understand that subsequent calls to jsonReadTypeNextIgnoreComma() are noops until the value has been processed.

Silence Coverity by assigning the actual type to a local variable so jsonReadTypeNextIgnoreComma() is only called once.

Also fix an adjacent comment typo.
2024-05-10 09:45:01 +09:30
David Steele
bb8988b551 Update storage (except remote) module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on.

The storage/remote module will be updated with the protocol module once a major waiting refactor has been committed.
2024-04-27 19:14:01 +10:00
David Steele
76bcb740b6
Add GCS batch delete support.
The GCS driver sent a single file delete request for each file while deleting a path. Depending on latency this could lead to rather long delete times, especially noticeable during expiration.

Improve GCS delete to use batches, which require multipart HTTP, so also add multipart HTTP infrastructure.
2024-04-27 15:42:10 +10:00
Michael Schout
e00e33b528
Dynamically find python in meson build.
This is better than requiring a python3 binary to be on the path because some installations might have, e.g. python3.9.

Also add the python3-distutils package to Debian builds to make this work.
2024-04-25 19:32:07 +10:00
David Steele
55e996912a Update meson minimum version to 0.47.
This allows enabling the check option for run_command, which automatically fails when the command fails.
2024-04-25 18:35:48 +10:00
Reid Thompson
4ea4e3f380
Update Fedora test image to Fedora 38.
This should have been done in 434938e3 but somehow it didn't happen.

Fedora 38 requires 2048 bit keys so update the VM builds to use them. Update the documentation to use 2048 bit keys. This is not technically required by this commit but it makes sense to do it now.

Also update the key location for the yum.p.o repository.

Lastly, shuffle test PostgreSQL versions since PostgreSQL 11 is not longer available in the yum.p.o repository.
2024-04-23 22:34:32 +10:00
David Steele
b40c2616c2 Update info module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on.
2024-04-21 13:16:53 +10:00
David Steele
c6fcc81db6 Update db/postgres modules to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 13:01:40 +10:00
David Steele
19411f39d2 Update config module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 12:06:39 +10:00
David Steele
0e36e1a2fc Update common module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-21 12:01:01 +10:00
David Steele
987e6e31f4 Move packager note about meson to new version.
Since there were some issues found with the meson install (7877983a, 7b95fd3b) it makes sense for any packagers who have not made the migration to hold off until the next release.

Move the note to the next release where hopefully all issues have been addressed.
2024-04-21 09:42:11 +10:00
David Steele
9546b9c5d0 Update command module to recent coding standards.
Add const as appropriate and avoid initializing variables if the variable will definitely be set later on or is immediately returned.
2024-04-20 14:19:27 +10:00
David Steele
fb22f04555
PostgreSQL 17 Support.
Add catalog version and WAL magic for PostgreSQL 17.
2024-04-18 10:56:24 +10:00
David Steele
c8cf8e1f2b Update contributor name. 2024-04-17 19:02:28 +10:00
udf2457
57731b6cd2
S3 SSE-C encryption support.
This feature (enabled with --repo-s3-sse-customer-key) provides an encryption key to encrypt the data after it has been transmitted to the server.

While not as secure as encrypting data before transmission (--repo-cipher-type), this may be useful in certain configurations.
2024-04-17 11:58:13 +10:00
David Steele
06d3cb767c Improve unit testing of valid page sizes in pg_control.
Spread the tests over more versions of PostgreSQL and improve/fix comments.
2024-04-17 11:43:41 +10:00
Michael Schout
7b95fd3bd2
Allow explicit disabling of optional dependencies in meson builds.
On some platforms, e.g. FreeBSD, there is a requirement to allow the user to disable support for features even when the required library is present.

Introduce tri-state options for the optional features: auto mimics the current behavior and is the default, enable requires libraries for the feature to be present, and disable disables the feature without checking the libraries.
2024-04-10 09:23:17 +10:00
David Steele
dab52739cd Fix comment typo. 2024-04-04 18:21:26 +11:00
David Steele
fec1b215e3 Remove test data files made obsolete by d41b21c8.
These should have been removed when the mock integration tests were removed.

Ideally we would also remove filecopy.table.bin but it serves to provide realistic page data for performance testing.
2024-03-30 10:43:05 +11:00
David Steele
9f5a97139f Allow strIdToLog() to output "null" values.
A valid StringId can never be zero so it more or less serves as a NULL value. In most cases zero will not be valid, but it is better to catch this condition with an assert rather than an error in logging.
2024-03-29 12:21:06 +11:00
David Steele
014e24889c Remove extra space before colons in meson.build files.
The spacing was not consistent so use the style that best matches our general coding standards.
2024-03-27 09:53:49 +11:00