1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4268 Commits

Author SHA1 Message Date
David Steele
6f250bf686 Fix groupId type in StorageInfo.
gid_t is the correct type but there was no bug because uid_t and gid_t have the same underlying type.
2023-05-06 12:15:21 +03:00
David Steele
ad9196cb49 Update Debian user guide to build on Ubuntu 20.04.
Ubuntu 18.04 will be EOL at the end of the month so update to a newer version.
2023-05-05 10:08:49 +03:00
David Christensen
023d3c99d8 Fix incorrect commit hash in 2f322e2a. 2023-05-04 08:52:31 +03:00
David Steele
2f322e2a49 Add efac3474 (remove double spacing) to .git-blame-ignore-revs. 2023-05-03 09:49:24 +03:00
David Steele
d49907239e Remove double spaces missed in 1bd5530a. 2023-05-03 09:37:59 +03:00
David Steele
efac3474cf Add 1bd5530a (remove double spacing) to .git-blame-ignore-revs. 2023-05-02 12:59:20 +03:00
David Steele
1bd5530a59 Remove double spaces from comments and documentation.
Double spaces have fallen out of favor in recent years because they no longer contribute to readability.

We have been using single spaces and editing related paragraphs for some time, but now it seems best to update the remaining instances to avoid churn in unrelated commits and to make it clearer what spacing contributors should use.
2023-05-02 12:57:12 +03:00
David Steele
5ea7b91bf9 Remove extraneous space from error messages. 2023-05-02 12:07:28 +03:00
David Steele
9637d94490 Use boolean variant constants where possible instead of redeclaring. 2023-05-02 09:37:42 +03:00
David Steele
822f2a5842
Finalize block incremental feature.
Remove beta status and update documentation to remove beta references and warnings.

The repo-block-* sub-options have been marked internal. Most users will be best off with the default behavior and we may still decide to change these options for remove them in the future.
2023-05-01 19:26:48 +03:00
David Steele
c510046dd7 Remove extraneous spaces in user guide. 2023-05-01 14:27:56 +03:00
David Steele
dd4e52679e
Removing chunking and block numbers from incremental block list.
These were intended to allow the block list to be scanned without reading the map but were never utilized. They were left in "just in case" and because they did not seem to be doing any harm.

In fact, it is better not to have the block numbers because this allows us set the block size at a future time as long as it is a factor of the super block size. One way this could be useful is to store older files without super blocks or a map in the full backup and then build a map for them if the file gets modified in a diff/incr backup. This would require reading the file from the full backup to build the map but it would be more space efficient and we could make more intelligent decisions about block size. It would also be possible to change the block size even if one had already been selected in a prior backup.

Omitting the block numbers makes the chunking unnecessary since there is now no way to make sense of the block list without the map. Also, we might want to build maps for unchunked block lists, i.e. files that were copied normally.
2023-04-27 23:29:12 +03:00
David Steele
3fc3690dd7
PostgreSQL 16 Support.
Add catalog version and WAL magic for PostgreSQL 16.

The GUC to force parallel mode has be renamed so update that in the tests.
2023-04-27 10:30:50 +03:00
David Steele
39059dc4e7 Remove extra linefeed. 2023-04-26 15:00:36 +03:00
David Steele
3ff88ffbb4
Avoid chown() on recovery files during restore.
The chown() was already skipped on the files restored from the repository but the same logic was not applied to the generated recovery files, probably because chown'ing a few recovery files does not have performance implications. Use the same logic for recovery files to determined if they need to be chown'd.

Ultimately this behavior is pretty hard to test, so add a fail safe into the Posix driver that will skip chown if the permissions are already as required.
2023-04-25 11:52:28 +03:00
David Steele
750ab8e55c Add MacOS back to CirrusCI.
9e29c01 removed MacOS testing due to breaking changes in the update to arm on the platform.

Update the scripts to correctly work with the version of Homebrew deployed with the arm images.
2023-04-21 16:21:10 +03:00
David Steele
23d7d67d49 Fix missing void parameter. 2023-04-21 15:08:59 +03:00
David Steele
f5e6bc2698
Allow page header checks to be skipped.
These checks cause false negatives for page checksum verification when the page is encrypted because pd_upper might end up as 0 in the encrypted data. This issue is rare but reproducible given a large enough cluster.

Make these checks optional, but leave them enabled by default.
2023-04-20 13:24:12 +03:00
David Steele
8240eb5da5 Autogenerate PostgreSQL versions.
This will make adding/removing versions of PostgreSQL more reliable.
2023-04-16 17:41:27 +03:00
David Steele
a05bf6bb15 Rename PG_VERSION_*_STR constants to PG_VERSION_*_Z.
This is more consistent with other zero-terminated string constants and also has the benefit of being shorter.
2023-04-16 17:32:24 +03:00
David Steele
75254c9285 Parameterize configLoad() as cfgLoadP().
There is one existing optional parameter and there are more to come.
2023-04-11 16:28:29 +04:00
David Steele
d6cb3de17a Update command/check module to recent coding standards.
Add const as appropriate and remove an unneeded declaration.
2023-04-11 14:50:59 +04:00
David Steele
a9c5cd9749 Update command/backup module to recent coding standards.
Add const as appropriate and avoid setting results to NULL if the result will definitely be set later on.
2023-04-11 12:46:00 +04:00
David Steele
f33e1cfb16 Add error retry detail for HTTP retries.
This should make it clearer when retries have happened and for how long.
2023-04-06 11:38:18 +04:00
David Steele
df419b34b5 Improve retry error messages.
Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code.

Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
David Steele
801e396dac Move error modules to common/error directory.
There are already enough files to warrant a separate directory and more are coming.

Also remove extraneous includes.
2023-04-06 10:38:49 +04:00
David Steele
ff98636e41 Update 32-bit CI host to Debian 10 from Ubuntu 18.04.
Ubuntu 18.04 will be EOL before the next release, so update to the oldest available Debian version.

Also fix one incorrect return value type, a test cast, and adjust some test timeouts.
2023-04-06 08:22:14 +04:00
David Steele
a9f39857cf Remove unused strOS parameter in Common::HostTest. 2023-04-05 11:24:53 +04:00
David Steele
8f7f73e4af Sleep using nanosleep() instead of select().
This is a safer way to sleep due to select's not-portable interaction with signals.

Based on https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a948e49e2ef11815be0b211723bfc5b53b7f75a8 from the PostgreSQL project.
2023-03-30 17:46:25 +05:00
David Steele
2cfbee903a Use Buffer object for page buffer in PageChecksum filter.
This avoids the object needing to support allocations.
2023-03-30 15:43:37 +05:00
David Steele
3dbf518c3a Convert XmlNode from allocation to object.
This makes memory accounting more accurate and objects that use Xml no longer need to allow allocations.
2023-03-30 14:50:46 +05:00
David Steele
047472144b Remove allocations from common crypto module.
The allocations are better done with a struct and Buffer object.
2023-03-30 14:42:20 +05:00
David Steele
84ca7b9b27 Cleanup for objects that do not require allocations.
A lot of these are left over from when object interfaces required allocations (changed in f6e30736 and 9ca9c8e4). Others are likely copy/paste errors.

This saves some space in the mem context and makes it clear that no allocations will be made.
2023-03-30 11:42:22 +05:00
David Steele
fe0fd71be2 Convert the result of zNewFmt() into an object.
The result is not intended to be freed directly so this makes memory tracking more accurate. Fix a few places where memory was leaking after a call to zNewFmt().

Also update an assert to make it clearer.
2023-03-30 10:00:58 +05:00
David Steele
295b53e845 Update Github actions to new versions.
These updates remove various deprecation warnings.
2023-03-28 16:24:43 +06:00
David Steele
b111599bad Simplify object creation with OBJ_NEW_BEGIN() macro.
Eliminate the boilerplate of declaring this and assigning memory to it, which is the same for the vast majority of object creations.

Keep the old version of the macro as OBJ_NEW_BASE_BEGIN() for a few exceptions in the core code and (mostly) in the tests.
2023-03-28 15:05:18 +06:00
David Steele
91f9301b9d Fix command/manifest unit test title. 2023-03-27 16:42:35 +06:00
David Steele
9ca9c8e4c9 Improve interface handling in remaining modules.
As in f6e30736, make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object. Allow exceptions to this general rule for objects that need to retain ownership of their interfaces.
2023-03-27 14:32:37 +06:00
David Steele
5f001248cc Fix indentation in config.yaml. 2023-03-25 14:59:47 +07:00
David Steele
8ff956ad7e Add lock module initialization.
Each call to lockAcquireP() passed enough information to initialize the lock system. This was somewhat inefficient and as locks become more complicated it will lead to more code duplication. Since a process can only take one type of lock it makes sense to do most of the initialization up front.

Also reduce the log level of lockRelease() since it is only called at exit and the lock will be released in any case.
2023-03-25 14:07:31 +07:00
David Steele
f1caecc4ff Convert lockAcquire() to lockAcquireP().
This makes a few current parameters optional and allows for more optional parameters with less code churn.
2023-03-24 10:34:42 +08:00
David Steele
c8ec114c8c Add reference filter and output to manifest command.
This allows the file list to be filtered by reference. The reference is output when it is not the default reference for the backup.
2023-03-21 12:29:45 +08:00
David Steele
1807bfcff5 Clear buffer limit when buffer is limited to allocated size.
This minor optimization automatically clears the limit flag when limit is set to allocated size. This has no impact on the current code but will simplify a future commit where a conditional bufLimitClear() is being used.
2023-03-21 12:22:47 +08:00
David Steele
5b5786c082 Begin v2.46 development. 2023-03-20 10:25:44 +08:00
David Steele
6ad79d16ca v2.45: Block Incremental Backup (BETA)
Bug Fixes:

* Skip writing recovery.signal by default for restores of offline backups. (Reviewed by Stefan Fercot. Reported by Marcel Borger.)

Features:

* Block incremental backup (BETA). (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)

Improvements:

* Keep only one all-default group index. (Reviewed by Stefan Fercot.)

Documentation Improvements:

* Add explicit instructions for upgrading between 2.x versions. (Contributed by Christophe Courtois. Reviewed by David Steele.)
* Remove references to SSH made obsolete when TLS was introduced.
2023-03-20 09:37:23 +08:00
David Steele
04d224b88e Add beta feature and documentation for block incremental backup. 2023-03-18 20:47:03 +08:00
David Steele
78c036efb7 Updates required for new manifest command to work remotely.
Additional options are required and the BlockChecksum must work remotely.
2023-03-16 17:46:32 +07:00
David Steele
505a639f1e Prototype manifest command.
Output a manifest in text or JSON format. Neither format is complete but they cover the basics.

In particular the manifest command outputs the complete block restore when the filter option is specified and the block delta when the pg option is also specified. This is non-destructive so it is safe to execute against a running cluster.
2023-03-14 21:56:05 +07:00
David Steele
c30d3e439b
Block incremental map fixes and improvements.
Bug Fixes:

* Remove the distinction between maps where super block size is equal to block size and maps where they are not. In practice, maps with equal blocks are now rare and most of the optimizations can be applied directly to super blocks where the blocks are equal. This fixes a bug where a map that was created with equal size blocks and then converted to differing block sizes would generate an invalid map.

* Free reads during restore to avoid running out of file handles.

Improvements:

* Store super block sizes in the block map. This allows the final block size to be removed from the block list and provides a more optimal restore and better potential for analysis.

* Always round the super block size up to the next block size. This makes the number of blocks per super block more predictable.

* Allow super block sizes to be changed at will in the map. The first case for this is to store the reduced super block size required when the last super block is short but it could be used to dynamically change the super block size to optimize compression.

* Store a block count rather than a list of blocks in a super block. Blocks must always be sequential, though there may be an offset to the first block in a super block. This saves 11-14% on space for checksum sizes 6-7.

* In the case that all the blocks for a super block are present, and there is no offset, the block size is omitted.
2023-03-14 17:48:25 +07:00
David Steele
5c1f78d4dd Fix typo in blockIncrProcess(). 2023-03-12 22:38:38 +07:00