1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

4548 Commits

Author SHA1 Message Date
David Steele
ff47450402
Update resume functionality for block incremental.
Resume was not updated for block incremental so block incremental files were always removed during a resume. Resume worked but was very inefficient with block incremental enabled.

Update resume to preserve block incremental files and add tests.
2024-03-12 12:41:58 +13:00
David Steele
e634fd85ce
Prevent invalid recovery when backup_label removed.
If backup_label is removed from a restored backup then PostgreSQL will instead use checkpoint information from pg_control to attempt (what is thinks is) crash recovery. This will nearly always result in a corrupt cluster because the checkpoint will not be from the beginning of the backup, and even if it is, the end point will not be specified, which could lead to recovery stopping too early.

To prevent this, invalidate the checkpoint LSN in pg_control on restore. If backup_label is removed then recovery will still fail because PostgreSQL will not be able to find the invalid checkpoint. The LSN of the checkpoint is not logged but it will be visible in pg_controldata output as 0/DEAD. This value is invalid because PostgreSQL always skips the first WAL segment when initializing a cluster.
2024-03-10 17:08:42 +13:00
David Steele
960b43589d Add validation for WAL segment size in pg_control.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being near the end and containing a limited number of discrete values.
2024-03-10 16:17:50 +13:00
David Steele
63541b2273 Add validation for page checksum version in pg_control.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being all the way at the end and being four bytes that can only have one of two values. Something more distinctive than 0 and 1 would be better, but this is what we have to work with.

Convert PgControl.pageChecksum to unsigned int and rename to PgControl.pageChecksumVersion and make all downstream changes required for the new datatype.
2024-03-10 15:50:10 +13:00
David Steele
7448fde157
Improved support for dual stack connections.
Connections are established using the "happy eyeballs" approach from RFC 8305, i.e. new addresses (if available) are tried if the prior address has already had a reasonable time to connect. This prevents waiting too long on a failed connection but does not try all the addresses at once. Prior connections that are still waiting are rechecked periodically if no subsequent connection is successful.

This improves substantially on 39bb8a0, which failed to take into account connection attempts that do not fail (but never connect) and use up all the available time.
2024-03-10 11:36:39 +13:00
David Steele
f287178b70
Set function/variable visibility to hidden.
This saves about 16KiB in the binary and reduces exported symbols by about 75%. All variables are still exported and any functions that are referenced by their pointers or extern'd but never used outside the module where they are defined.

In addition to modest space savings, this should also increase performance a bit since the compiler can simplify calls to these functions and load the binary should also be a little faster.

The GCC documentation does not make it clear that visibility can be used with variables, but it certainly makes a difference in the binary size, so something is happening. Other sources on the internet suggest that visibility can be used with variables. Clearly exports are not affected, but there may be some other optimization happening.
2024-03-10 11:09:13 +13:00
David Steele
e3d9df3ae9
Make meson the primary build system.
Meson has a lot of advantages over autoconf/make, primarily in ease-of-use and performance. Make meson the only build system used for testing and building the Debian documentation, but leave the RHEL documentation using autoconf/make for now so it gets some testing.
2024-03-10 10:53:31 +13:00
David Steele
c64cd8e019 Disable arm64 test in CirrusCI.
There seems to be a shortage of arm64 hosts because queue times have been steadily increasing over the last few weeks. It can now take several hours to get an arm64 test queued, which makes it difficult to get development done.

Disable for the time being and hope the resource issue gets resolved in the future.
2024-03-10 10:43:24 +13:00
David Steele
eda7706f53 Make valgrind libssh2 leak check less specific.
The leak kind is usually definite but sometimes flaps to possible. For stability purposes accept any leak kind.

Note that this is a leak in a specific version of libssh2 and not a bug in pgBackRest.
2024-03-10 09:40:13 +13:00
David Steele
dddcbcd8e9
Limit resume functionality to full backups.
Resume does not work correctly with delta diff/incr backups because the presence of a reference causes it to remove the file with the idea that it can just be referenced again. This is true for timestamp-based backups but for deltas all existing files need to be rechecked (which requires a reference).

This is fixable but not without significant effort and new tests and it calls into question the usefulness of non-full resumes. For diff/incr, if the file was changed since the prior backup there is a good chance it will be modified again before the resume occurs.

In order to keep this feature as useful as possible for the most valuable case, limit resumes to full backups.
2024-03-10 09:32:55 +13:00
David Steele
9d91d1b2f8
Detect files that have not changed during non-delta incremental backup.
02eea55 added code to load a buffer of data from a file being backup up to detect files that have been truncated to zero after manifest generation. This mechanism can also be used to detect files that have not changed since the prior backup.

If the result of the file copy fits into a single buffer, then the size and checksum can be compared to the prior file before anything gets stored. If the file matches then it is referenced to the file in to prior backup.

The size that can be compared for normal copies is limited by the buffer size but for block incremental it works with any size file since there is no output from block incremental when the file is identical.
2024-03-08 15:07:43 +13:00
David Steele
cf17515e40
Improve archive-push WAL segment queue handling.
Infer the size of all WAL segments from the size of the first segment rather than getting info for all segments (up to queue size). If the segments are not the same size then there are larger issues than the WAL queue.
2024-03-08 12:34:11 +13:00
David Steele
4387250f2e
Improve sort comparators.
Improve sort comparators to use branchless comparisons when possible and avoid using subtraction. Only one comparator was using subtraction and it appears there was no overflow risk since the values were pretty small.

Inspired by https://www.postgresql.org/message-id/CA%2B14426g2Wa9QuUpmakwPxXFWG_1FaY0AsApkvcTBy-YfS6uaw%40mail.gmail.com.
2024-03-08 10:07:03 +13:00
David Steele
e00bfe2d2c
Fix performance regression in storage list.
storageListP() returns a list of entries in a path and should not need to stat/head, etc. in order to get more detailed info. This was broken by 75623d4 which failed to set the level correctly.

Set the correct level and update tests.

There's no easy way to directly test for a regression here but the SFTP tests will fail if more detailed info is requested since it would require script changes.
2024-03-08 09:50:20 +13:00
David Steele
794c577130 Migrate integration tests to C.
The Perl integration tests were migrated as faithfully as possible, but there was some cruft and a few unit tests that it did not make sense to migrate.

Also remove all Perl code made obsolete by this migration.

All unit, performance, and integration tests are now written in C but significant parts of the test harness remain to be migrated.
2024-03-06 11:00:09 +13:00
David Steele
7f1bb3a051 Remove obsolete valgrind suppression. 2024-03-05 17:45:25 +13:00
David Steele
cb58bdc975 Move assert in cmdServerSigChild().
It is possible for the server to receive notifications for children that were not spawned by it, e.g. when running as the root process of a container.

Only assert when the child process is found in our process list. Otherwise, the code may be something unexpected.

Since this is an assert there should not be any real world consequences either way, but the issue has caused problems in testing.
2024-03-04 18:57:19 +13:00
David Steele
07051347fb Update Cirrus CI BSD image to 13.3. 2024-03-04 18:14:41 +13:00
David Steele
f7a7ab16c9
Skip zero-length files for block incremental delta restore.
a42614e introduced the capability to preserve smaller than expected files for block incremental restore delta, but failed to take into account that zero-length files are both useless and cause the block checksum filter to error.

Fix this by skipping zero-length files during block incremental restore delta.
2024-03-02 12:29:10 +13:00
David Steele
6c45b57fa8 Add execOne() to simplify exec for build, documentation, and testing.
The core Exec object is efficient but geared toward the specific needs of core and not ease-of-use as required for build, documentation, and testing.

execOne() works similarly to system() except that it automatically redirects stderr to stdout and captures the output.
2024-02-24 11:22:48 +13:00
David Steele
6356a2b76c Disable log expect testing for non-unit tests.
This has never been a problem for performance tests since they do not call functions that log at info level or above, but the upcoming integration tests may do so. In any case it is better to disable this functionality outside of unit tests.
2024-02-17 19:16:39 +13:00
David Steele
d41b21c8f7 Remove mock integration tests.
These tests have not been maintained for several years, i.e. no tests for new features have been added. They are highly duplicative of the unit tests but do have the advantage of mixing in different storage drivers. They were allowed to remain because they were not doing any harm even if they were probably not doing any good.

However, the real integration tests (that run directly against PostgreSQL) also test storage drivers and have been updated with new features over time. The real integration tests are now being migrated to C and as part of that effort the mock integration tests need to be removed or migrated, and they do not provide enough value to migrate.

Remove all mock integration tests and a leftover Perl performance test.
2024-02-17 10:52:37 +13:00
David Steele
68351785f6
Update references to recovery.conf.
Update references to recovery.conf to include postgresql.auto.conf used in newer versions.

Also update a broken recovery URL and point it to the current version (with a hint to select the proper version of PostgreSQL).
2024-02-16 11:37:31 +13:00
David Steele
273d5050f8 Add asserts to default switch cases in common/type/variant module.
This matches how switch is used in other parts of the code and prevents invalid values from being processed normally.
2024-02-07 11:01:55 -03:00
David Steele
ad8e92a9ac Remove unneeded backupTimeStart parameter from testBackupValidate().
5ce8b9df added this parameter, but it is better to get the value from the manifest, which is already present.
2024-02-05 10:10:59 -03:00
David Steele
45f070aa9a Add release instructions to update PostgreSQL ecosystem wiki. 2024-01-25 10:41:20 -03:00
David Steele
5ce8b9dfae Simplify output in command/backup unit tests.
The output combined a representation of the files/paths/links in the manifest along with output of what was validated on disk. This was redundant and made maintenance of the tests difficult, especially with all the quoting in the manifest output (which also made it hard to search for output).

Instead use primarily the output created during validation and add fields from the manifest that were missing. Exclude paths from the output when there are files in the path since the path is then implied.

One major change here is that checksums are no longer output. This makes it easier to write tests that work on multiple architectures and all checksums are already verified during validation.
2024-01-24 14:54:28 -03:00
David Steele
1a0cc96c5a Migrate man page generation to C. 2024-01-23 09:56:26 -03:00
David Steele
676700d8ca
Add detailed backtrace to autoconf/make build.
c8264291 added libbacktrace to the meson build (not used in production yet), but held off on adding it to autoconf/make before more performance testing was done.

Performance tests show there is no noticeable cost to adding libbacktrace, so add it to get more detail error stack traces.
2024-01-22 14:17:09 -03:00
David Steele
0aaa0772f5 Fix help not displaying help for the help command.
'pgbackrest help help' just displayed the help overview, rather than display help for the help command.

Fix this by making sure the command is set and routing correctly in the help command.
2024-01-22 12:33:40 -03:00
David Steele
68db3075d7
Allow --version and --help for version and help.
It is a bit confusing that --help and --version do not work like most command-line programs. For example, git allows either --help or help.

Make these work by making them shortcuts (not actual options) to the applicable commands.

The user will still need to use help (not --help) to get help on specific commands/options, but at least they can get to the main help (which will tell them this) via --help.
2024-01-22 12:00:13 -03:00
David Steele
db5bcff3b4 Update config.guess and config.sub to latest versions. 2024-01-22 10:11:48 -03:00
David Steele
c618fd3b09 Begin v2.51 development. 2024-01-22 10:09:38 -03:00
David Steele
1add35624d v2.50: Performance Improvements and Bug Fixes
Bug Fixes:

* Fix short read in block incremental restore. (Reviewed by Stephen Frost, Brent Graveland. Reported by Adol Rodriguez, Brent Graveland.)
* Fix overflow suppressing backup progress in info output. (Fixed by Robert Donovan. Reviewed by Joe Wildish.)

Improvements:

* Preserve partial files during block incremental delta restore. (Reviewed by Stephen Frost.)
* Add support for alternate compile-time page sizes. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
* Skip files truncated during backup when bundling. (Contributed by Georgy Shelkovy. Reviewed by David Steele.)
* Improve SFTP storage error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
2024-01-22 09:54:59 -03:00
David Steele
fd8974914d Cleanup tablespace tests in command/backup module.
Tablespaces were enabled for tests that did not have tablespaces, resulting in tablespace_map being present in backups even when it was not needed.

Instead of specifying if tablespaces are present, automatically detect tablespaces in hrnBackupPqScript().
2024-01-12 13:55:08 -03:00
David Steele
3926dd346e Update LICENSE.txt and PostgreSQL copyright for 2024. 2024-01-04 14:55:44 -03:00
David Steele
a484862763 Remove installation of old yum.p.o repository key. 2024-01-04 14:53:44 -03:00
David Steele
55f2248963 Break up scripts in db unit test module.
Long scripts followed by a number of tests are really challenging to debug and update.

Instead, break up the scripts to be inline with the tests that they drive. This should make maintenance of the tests much simpler.
2024-01-03 18:15:50 -03:00
Reid Thompson
7c17eec3db
Improve SFTP storage error messages.
Use storageSftpEvalLibSsh2Error() in more locations to provide better error information. Also add storageSftpLibSsh2SessionLastError() for the same reason.
2024-01-03 13:01:50 -03:00
David Steele
802ae79148 Remove FreeBSD 12 and add FreeBSD 14 to Cirrus CI.
FreeBSD 12 is now EOL.

Also update the image version for FreeBSD 13.
2024-01-03 12:43:50 -03:00
Reid Thompson
8a8cfba62b Remove resolved comment. 2023-12-27 12:53:53 -03:00
David Steele
f4a4af299b
Fix short read in block incremental restore.
During restore it is possible to read all the blocks out of a compressed super block without reading all the input. This is because the compression format may have some trailing bytes that are not required for decompression but are required to indicate that data has ended. If a buffer aligned with the compressed data in a certain way, these last bytes might not be read.

Explicitly read out any final bytes at the end of each super block to handle this case. This should always result in no additional data out and we check for that, but it does move the read position to the beginning of the next compressed super block so decompression can begin without error.
2023-12-27 12:49:47 -03:00
David Steele
c47b39acf6 Fix incorrect test comment.
Left over from an older implementation.
2023-12-27 12:39:45 -03:00
David Steele
4324b568a9 Move block testBlockDelta() to harness module.
This makes the function available to other test modules.

Also rename to hrnBlockDeltaRender().
2023-12-26 21:07:56 -03:00
David Steele
9049fec2c0 Refactor skip files truncated during backup when bundling.
Refactor 02eea555 to always close the file immediately on EOF and use backupCopyResultCopy to continue processing. Closing the file immediately saves a later EOF check and is friendlier to added logic in this area. Using backupCopyResultCopy to continue is clearer also makes it easier to add new logic.

Also store zero checksum so the bulk of results collection can be moved within the copy block.
2023-12-22 13:16:45 -03:00
David Steele
c8795094d4 Allow const checksum buffers to be returned from backupFile().
This allows less duplication of buffers.

For delta check return file->pgFileSize/file->pgFileChecksum instead of pgTestSize/pgTestChecksum since this saves one buffer duplication and we know these values are equal since we just checked them.

Also add an assert to ensure copyChecksum is valid relative to size.
2023-12-22 12:48:01 -03:00
David Steele
4f760df417
Do not preserve block incremental if file is less than prior block size.
If a file stored with block incremental shrinks below the prior block size then the map is useless and the entire file needs to be stored again.

In this case use the new block incremental values (even if none) rather than preserving the old ones.
2023-12-22 00:59:12 -03:00
David Steele
3cd8249dba Update warning for backup resume invalid repo file.
This warning has had a note since the C migration that it should be moved below the backup file log message, so do that.

Also update the warning message a bit to correct for tense. This message was likely in a different place originally.
2023-12-21 15:28:37 -03:00
David Steele
701865eca1 Refactor backupFile() to remove backupCopyResultReCopy.
Having two enum values for file copy makes things a bit more complicated than they need to be (especially in an upcoming commit).

Instead add a flag to indicate that the repository file was invalid since the only purpose is to trigger a warning message.
2023-12-21 15:20:03 -03:00
David Steele
a42614e8f3
Preserve partial files during block incremental delta restore.
Previously files that were smaller than the expected size were not preserved for block incremental, even though it is possible that block incremental could make use of a partial file.

One example is when a restore encounters an error. On retry the partial file can be used as a starting point rather than copying again from the beginning. Another example is restoring a backup where a file is larger than what already exists in the data directory.

Preserve any size file when block incremental will be used for the delta in order to reuse partial files when possible. If the file is smaller than expected then disable the whole-file checksum to reduce overhead.
2023-12-21 15:08:07 -03:00