1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2523 Commits

Author SHA1 Message Date
David Steele
dd4e52679e
Removing chunking and block numbers from incremental block list.
These were intended to allow the block list to be scanned without reading the map but were never utilized. They were left in "just in case" and because they did not seem to be doing any harm.

In fact, it is better not to have the block numbers because this allows us set the block size at a future time as long as it is a factor of the super block size. One way this could be useful is to store older files without super blocks or a map in the full backup and then build a map for them if the file gets modified in a diff/incr backup. This would require reading the file from the full backup to build the map but it would be more space efficient and we could make more intelligent decisions about block size. It would also be possible to change the block size even if one had already been selected in a prior backup.

Omitting the block numbers makes the chunking unnecessary since there is now no way to make sense of the block list without the map. Also, we might want to build maps for unchunked block lists, i.e. files that were copied normally.
2023-04-27 23:29:12 +03:00
David Steele
3fc3690dd7
PostgreSQL 16 Support.
Add catalog version and WAL magic for PostgreSQL 16.

The GUC to force parallel mode has be renamed so update that in the tests.
2023-04-27 10:30:50 +03:00
David Steele
3ff88ffbb4
Avoid chown() on recovery files during restore.
The chown() was already skipped on the files restored from the repository but the same logic was not applied to the generated recovery files, probably because chown'ing a few recovery files does not have performance implications. Use the same logic for recovery files to determined if they need to be chown'd.

Ultimately this behavior is pretty hard to test, so add a fail safe into the Posix driver that will skip chown if the permissions are already as required.
2023-04-25 11:52:28 +03:00
David Steele
f5e6bc2698
Allow page header checks to be skipped.
These checks cause false negatives for page checksum verification when the page is encrypted because pd_upper might end up as 0 in the encrypted data. This issue is rare but reproducible given a large enough cluster.

Make these checks optional, but leave them enabled by default.
2023-04-20 13:24:12 +03:00
David Steele
8240eb5da5 Autogenerate PostgreSQL versions.
This will make adding/removing versions of PostgreSQL more reliable.
2023-04-16 17:41:27 +03:00
David Steele
a05bf6bb15 Rename PG_VERSION_*_STR constants to PG_VERSION_*_Z.
This is more consistent with other zero-terminated string constants and also has the benefit of being shorter.
2023-04-16 17:32:24 +03:00
David Steele
75254c9285 Parameterize configLoad() as cfgLoadP().
There is one existing optional parameter and there are more to come.
2023-04-11 16:28:29 +04:00
David Steele
f33e1cfb16 Add error retry detail for HTTP retries.
This should make it clearer when retries have happened and for how long.
2023-04-06 11:38:18 +04:00
David Steele
df419b34b5 Improve retry error messages.
Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code.

Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
David Steele
801e396dac Move error modules to common/error directory.
There are already enough files to warrant a separate directory and more are coming.

Also remove extraneous includes.
2023-04-06 10:38:49 +04:00
David Steele
ff98636e41 Update 32-bit CI host to Debian 10 from Ubuntu 18.04.
Ubuntu 18.04 will be EOL before the next release, so update to the oldest available Debian version.

Also fix one incorrect return value type, a test cast, and adjust some test timeouts.
2023-04-06 08:22:14 +04:00
David Steele
a9f39857cf Remove unused strOS parameter in Common::HostTest. 2023-04-05 11:24:53 +04:00
David Steele
047472144b Remove allocations from common crypto module.
The allocations are better done with a struct and Buffer object.
2023-03-30 14:42:20 +05:00
David Steele
84ca7b9b27 Cleanup for objects that do not require allocations.
A lot of these are left over from when object interfaces required allocations (changed in f6e30736 and 9ca9c8e4). Others are likely copy/paste errors.

This saves some space in the mem context and makes it clear that no allocations will be made.
2023-03-30 11:42:22 +05:00
David Steele
fe0fd71be2 Convert the result of zNewFmt() into an object.
The result is not intended to be freed directly so this makes memory tracking more accurate. Fix a few places where memory was leaking after a call to zNewFmt().

Also update an assert to make it clearer.
2023-03-30 10:00:58 +05:00
David Steele
b111599bad Simplify object creation with OBJ_NEW_BEGIN() macro.
Eliminate the boilerplate of declaring this and assigning memory to it, which is the same for the vast majority of object creations.

Keep the old version of the macro as OBJ_NEW_BASE_BEGIN() for a few exceptions in the core code and (mostly) in the tests.
2023-03-28 15:05:18 +06:00
David Steele
91f9301b9d Fix command/manifest unit test title. 2023-03-27 16:42:35 +06:00
David Steele
9ca9c8e4c9 Improve interface handling in remaining modules.
As in f6e30736, make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object. Allow exceptions to this general rule for objects that need to retain ownership of their interfaces.
2023-03-27 14:32:37 +06:00
David Steele
8ff956ad7e Add lock module initialization.
Each call to lockAcquireP() passed enough information to initialize the lock system. This was somewhat inefficient and as locks become more complicated it will lead to more code duplication. Since a process can only take one type of lock it makes sense to do most of the initialization up front.

Also reduce the log level of lockRelease() since it is only called at exit and the lock will be released in any case.
2023-03-25 14:07:31 +07:00
David Steele
f1caecc4ff Convert lockAcquire() to lockAcquireP().
This makes a few current parameters optional and allows for more optional parameters with less code churn.
2023-03-24 10:34:42 +08:00
David Steele
c8ec114c8c Add reference filter and output to manifest command.
This allows the file list to be filtered by reference. The reference is output when it is not the default reference for the backup.
2023-03-21 12:29:45 +08:00
David Steele
1807bfcff5 Clear buffer limit when buffer is limited to allocated size.
This minor optimization automatically clears the limit flag when limit is set to allocated size. This has no impact on the current code but will simplify a future commit where a conditional bufLimitClear() is being used.
2023-03-21 12:22:47 +08:00
David Steele
6ad79d16ca v2.45: Block Incremental Backup (BETA)
Bug Fixes:

* Skip writing recovery.signal by default for restores of offline backups. (Reviewed by Stefan Fercot. Reported by Marcel Borger.)

Features:

* Block incremental backup (BETA). (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)

Improvements:

* Keep only one all-default group index. (Reviewed by Stefan Fercot.)

Documentation Improvements:

* Add explicit instructions for upgrading between 2.x versions. (Contributed by Christophe Courtois. Reviewed by David Steele.)
* Remove references to SSH made obsolete when TLS was introduced.
2023-03-20 09:37:23 +08:00
David Steele
78c036efb7 Updates required for new manifest command to work remotely.
Additional options are required and the BlockChecksum must work remotely.
2023-03-16 17:46:32 +07:00
David Steele
505a639f1e Prototype manifest command.
Output a manifest in text or JSON format. Neither format is complete but they cover the basics.

In particular the manifest command outputs the complete block restore when the filter option is specified and the block delta when the pg option is also specified. This is non-destructive so it is safe to execute against a running cluster.
2023-03-14 21:56:05 +07:00
David Steele
c30d3e439b
Block incremental map fixes and improvements.
Bug Fixes:

* Remove the distinction between maps where super block size is equal to block size and maps where they are not. In practice, maps with equal blocks are now rare and most of the optimizations can be applied directly to super blocks where the blocks are equal. This fixes a bug where a map that was created with equal size blocks and then converted to differing block sizes would generate an invalid map.

* Free reads during restore to avoid running out of file handles.

Improvements:

* Store super block sizes in the block map. This allows the final block size to be removed from the block list and provides a more optimal restore and better potential for analysis.

* Always round the super block size up to the next block size. This makes the number of blocks per super block more predictable.

* Allow super block sizes to be changed at will in the map. The first case for this is to store the reduced super block size required when the last super block is short but it could be used to dynamically change the super block size to optimize compression.

* Store a block count rather than a list of blocks in a super block. Blocks must always be sequential, though there may be an offset to the first block in a super block. This saves 11-14% on space for checksum sizes 6-7.

* In the case that all the blocks for a super block are present, and there is no offset, the block size is omitted.
2023-03-14 17:48:25 +07:00
David Steele
d7704a8bc8 Move backup pq test script generation to backup harness.
This allows scripted online backups to be used by other test modules.
2023-03-11 14:04:43 +07:00
David Steele
19f3a1d304 Add missing header. 2023-03-10 16:08:30 +07:00
David Steele
24f725212d Add beta feature infrastructure.
This allows options to be marked as beta, which will require that the --beta option be supplied to prevent accidental usage of a beta feature.

The online and command-line documentation also show warnings when options are beta.
2023-03-10 15:30:27 +07:00
David Steele
6b409d049e Update default block size and super block values based on testing.
Block sizes are incremented when the size of the map becomes as large as a single block. This is arbitrary but it appears to give a good balance of block size vs map size.

The full backup super block size is set to minimize loss of compression efficiency since most blocks in the database will likely never be modified. For diff/incr backup super blocks, a smaller size is allowable since only modified blocks are stored. The overall savings of not storing unmodified blocks offsets the small loss in compression efficiency due to the smaller super block and allows more granular fetches during restore.
2023-03-10 14:01:38 +07:00
David Steele
1119a53539 Rename BlockHash to BlockChecksum.
Checksum is the generally used terminology in the code base, even when a hash is being used as a checksum.
2023-03-09 11:04:03 +07:00
David Steele
6252c0e448 Exclude backup set size from info for block incremental backups.
As calculated this size is not correct since it does not include the parts of prior block incrementals that are required to make the current block incremental valid. At best this could be approximated and the resulting values might be very confusing.

For now, at least, exclude this metric for block incremental backups.
2023-03-09 10:30:57 +07:00
David Steele
210bed4511 Use xxHash instead of SHA-1 for block incremental checksums.
xxHash is significantly faster than SHA-1 so this helps reduce the overhead of the feature.

A variable number of bytes are used from the xxHash depending on the block size with a minimum of six bytes for the smallest block size. This keeps the maps smaller while still providing enough bits to detect block changes.
2023-03-09 10:02:04 +07:00
David Steele
8b5153ad21
Block-level incremental backup super blocks.
Small blocks sizes can lead to reduced compression efficiency, so allow multiple blocks to be compressed together in a super block. The disadvantage is that the super block must be read sequentially to retrieve blocks. However, different super block sizes can be used for different backup types, so the full backup super block sizes are large for compression efficiency and diff/incr are smaller for retrieval efficiency.
2023-03-09 09:39:54 +07:00
Stefan Fercot
740c2258e3
Add pg-version-force option for fork integration.
Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on.

This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.
2023-03-09 08:23:15 +07:00
David Steele
2fa7e53c5d
Skip writing recovery.signal by default for restores of offline backups.
When restoring an offline backup on PostgreSQL >= 12, skip writing recovery.signal by default since this will error if the backup was made with wal_level=minimal. If the user explicitly sets the type option to something other than none, then write recovery.signal as usual since it is possible to do Point-In-Time-Recovery from an offline backup as long as wal_level was not minimal.
2023-03-08 19:05:23 +07:00
David Steele
7e5adc0359 Use raw compression/encryption to bundling and block incremental backup.
Raw encryption was already being used for block incremental. This commit adds raw compression to block incremental where possible (see da918587).

Raw compression/encryption is also added to bundling for a backup set when block incremental is enabled on the full backup. This prevents a break in backward compatibility since block incremental is not backward compatible.
2023-03-07 18:46:24 +07:00
David Steele
da91858702 Add optional raw format for compression types.
Raw format saves 12 bytes of header for gzip and 4 bytes of checksum for lz4 (plus CPU overhead). This may not seem like much, but over millions of small files or incremental blocks can really add up. Even though it may be a relatively small percentage of the overall backup size it is still objectively a large amount of data.

Use raw format for protocol compression to exercise the feature.

Raw compression format will be added to bundling and block incremental in a followup commit.
2023-03-07 18:31:17 +07:00
David Steele
f6e307365f Improve interface handling in storage module.
Make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object.

The prior method was more efficient when mem contexts had a much higher cost. Now mem contexts are cheap so it makes more sense to structure the objects in a way that works better with mem context auditing. This also means the mem context does not need to be stored separately since it can be extracted directly from the interface object.

There are other areas that need to get the same improvement before the specialized objMoveContext() and objFreeContext() functions can be removed.
2023-03-07 11:27:41 +07:00
David Steele
0818601c05 Set online flag in manifest in command/restore unit tests.
This flag does not currently affect restore behavior but it will in an upcoming commit. Set the flag here to simplify the test diff in the upcoming commit.
2023-03-06 09:49:33 +07:00
David Steele
c656669ac2 Allow control version and WAL magic to be overridden in test harness.
This makes it easier to write tests for invalid control version/WAL magic.

Also add HRN_PG_WAL_TO_BUFFER() to simplify generation of WAL headers.
2023-03-04 12:50:02 +07:00
David Steele
1648c133d6
Keep only one all-default group index.
It is possible for a group index to be created for an option that is later found to not meet dependencies. In this case all values would be default leading to a phantom group, which can be quite confusing.

Remove group indexes that are all default (except the final one) and make sure the key for the final all default group index is 1.
2023-03-04 12:45:08 +07:00
David Steele
16ac5ee8d3 Rename block incremental manifest keys.
Since the keys need to be read/written in order, these keys make the logic a bit simpler.
2023-02-26 16:13:44 +07:00
David Steele
a9867cb0b8 Add repo-block-age-map and repo-block-size-map options.
Make these options configurable. This is primarily for testing purposes so the new options will be kept internal.
2023-02-26 14:49:34 +07:00
David Steele
dffc933384 Rename DeltaMap to BlockHash.
This more accurately describes what the object does.
2023-02-13 09:17:30 +07:00
David Steele
779efe0d7a Consistently declare block incremental size as size_t.
The block is often allocated in memory so size_t makes more sense than uint64_t.
2023-02-09 13:01:56 +07:00
David Steele
089fae035b Add block incremental to real/all test output. 2023-02-07 14:09:50 +07:00
David Steele
31cad5e09e Check for stray execute permissions in test.pl --code-format.
Sometimes these can get set while updating permissions from with a VM or container.

Also fix a few permissions that were not correct.
2023-02-01 12:57:04 +07:00
David Steele
c5907a2e71 Remove references to SSH made obsolete when TLS was introduced.
Also remove details about SSH compression that are not helpful.
2023-01-31 08:28:32 +07:00
David Steele
ce0ea2cfab Use uncrustify for code formatting.
The code is not completely reflowed yet so there are some cases that uncrustify will not catch. The formatting will be improved over time.

Some block of code require special formatting so have been surrounded with the {uncrustify-off}/{uncrustify-on} markers. These exceptions should be kept to a minimum.

Add --code-format (to reformat code) and --code-format-check (to check formatting) to test.pl.

Add a CI test that will check code formatting. Code must be correctly formatted before it can be merge to integration.

Add documentation to the coding standards for code formatting.
2023-01-30 12:22:41 +07:00
David Steele
d4070c9064 Reformat code with uncrustify.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:

* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.

Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.

Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
David Steele
b2202c36d9 Fix formatting errors.
Errors in our current (manually-maintained) code format discovered by uncrustify.
2023-01-30 11:16:31 +07:00
David Steele
f91af30584 Add FN_PRINTF() macro.
Bring the format(printf) attribute in line with the FN_NO_RETURN and FN_INLINE_ALWAYS macros.

This is simpler to read and can be customized for different compilers.
2023-01-30 10:51:48 +07:00
David Steele
f8ea51bf8f Refactor stackTraceToZ() to avoid multiple versions of the function.
stackTraceToZ() was split this way in c8264291 to allow complete coverage. 0becb6da added a shim to improve coveage but missed simplifying the function.
2023-01-30 10:27:17 +07:00
David Steele
053468bfb1 v2.44: Remove PostgreSQL 9.0/9.1/9.2 Support
Improvements:

* Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
* Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
* Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)

Documentation Improvements:

* Add warning about enabling "hierarchical namespace" on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
* Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
* Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
* Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
* Add dark mode to the website. (Suggested by Stephen Frost.)
2023-01-30 09:15:44 +07:00
David Steele
a28f3d49c2 Add --no-back-trace option to test.pl.
Running valgrind and backtrace together has been causing tests to timeout in CI, mostly likely due to limited resources. This has not been a problem in normal development environments.

Since it is still important to run backtraces for debugging, split the u22 test that was doing all this work to run coverage and backtrace together and valgrind-only as a separate test. As a bonus these tests run faster separately and since they run in parallel the total execution time is faster.
2023-01-28 18:55:53 +07:00
David Steele
16c625353d Skip test modules in coverage report.
Coverage of the test code is useful during development but it does not seem worth adding it to the core code coverage report.
2023-01-28 16:22:04 +07:00
David Steele
d5ef4ce747 Fix permissions on source files. 2023-01-28 11:01:29 +07:00
David Steele
1da2666a9e Add manifest test harness.
These macros make adding paths/files/etc to a manifest simpler and easier to read.
2023-01-21 14:03:27 +07:00
David Steele
912eec63bb
Block-level incremental backup.
The primary goal of the block incremental backup is to save space in the repository by only storing changed parts of a file rather than the entire file. This implementation is focused on restore performance more than saving space in the repository, though there may be substantial savings depending on the workload.

The repo-block option enables the feature (when repo-bundle is already enabled). The block size is determined based on the file size and age. Very old or very small files will not use block incremental.
2023-01-20 16:48:57 +07:00
David Steele
008a18555c Fix comment indentation and wrapping. 2023-01-20 15:44:01 +07:00
David Steele
bfc9a5c33e Fix incorrect FUNCTION_TEST_END() in hrnPgWalToBuffer(). 2023-01-14 18:01:48 +07:00
David Steele
34e4835ff3
Refactor common/ini module to remove callbacks and duplicated code.
The callbacks in iniLoad() made the downstream code more complicated than it needed to be so use an iterator model instead.

Combine the two functions that were used to load the ini data to remove code duplication. In theory it would be nice to use iniValueNext() in the config/parse module rather than loading a KeyValue store but this would mean a big change to the parser, which does not seem worthwhile at this time.
2023-01-12 21:24:28 +07:00
David Steele
9ca492cecf Audit mem contexts returned from functions into the calling context.
It is possible for functions to accidentally leak child contexts into the calling context, which may use a lot of memory depending on the use case and where it happens.

Use the function return type to determine what should be returned and error when something else is returned. Add FUNCTION_AUDIT_*() macros to handle exceptions.

This checking is only performed during unit tests on the code being covered by the specific unit test.

Note that this does not work yet for memory allocations, i.e. memNew(). These are pretty rare so are not as much of an issue and they can be added in the future.
2023-01-12 17:36:57 +07:00
David Steele
de1dfb66ca Refactor logging functions to never allocate memory.
Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case.

This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit.

Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
David Steele
0becb6da31 Enhance libbacktrace to handle incomplete stack traces.
This fills in backtrace info at the bottom of the call stack when the stack trace is incomplete due to testing. This does not affect release builds, which is why it did not make the first cut, but it turns out to be useful for testing and barely changes the release code (when we do release this).

The recursion test in common/error was simplified because it would now return a very large trace.
2023-01-12 10:22:26 +07:00
David Steele
9347313cbf Fix error detail output to stderr.
The error detail should be output when the error is an assert (this part was working) or the log level is at least debug. In cases where log-level-console was at least debug but log-level-stderr was not the detail was lost.

Improve the range checking to output error detail to stderr when log-level-console is at least debug.
2023-01-11 19:16:31 +07:00
David Steele
c826429123 Improve libbacktrace implementation.
The libbacktrace feature has not been working since the move to meson because libbacktrace detection was not added to the meson build. Add libbacktrace to meson and improve the feature so that it can be compiled into release builds.

The prior implementation fetched line numbers with each stack trace push. Not only was this slow but it missed any functions that were not being tracked on our stack.

Instead just examine the backtrace when an error happens and merge it with the info we have on our stack. If the backtrace is not available then the output remains as before.

Also remove --backtrace from test.pl since the library is now auto-detected.

Leave this library out of the production build for now to give it a little time to shake out in testing.
2023-01-11 11:19:26 +07:00
David Steele
6633ccd18d Do not output stderr on unit/integration test errors.
Since stderr is being redirected to stdout this results in duplication of the error output.
2023-01-05 14:03:43 +07:00
David Steele
4429bc82f5 Add unit tests for the unit test build code.
When this code was migrated to C the unit tests were not included because there were more important priorities at the time.

This also requires some adjustments to coverage because of the new code location.
2023-01-05 12:59:06 +07:00
David Steele
a5499afabc Add CHECK_FMT() macros and tests for CHECK*() macros.
The CHECK_FMT() macro will be used in some upcoming tests but could also be useful in the core code.

The CHECK*() macros never had tests so add them.
2023-01-05 12:44:00 +07:00
David Steele
d51a86c621 Add BUFFER_EXTERN() and cleanup hash zero constants.
BUFFER_EXTERN() provides a clean way to create buffer constants.

Convert HASH_TYPE_SHA256_ZERO_STR to HASH_TYPE_SHA256_ZERO_BUF to be consistent with HASH_TYPE_SHA1_ZERO_BUF.
2023-01-02 17:30:57 +07:00
David Steele
f018912908 Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN.
This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can).

Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
David Steele
4fb8a0ecdd Add meson unity build and tests.
This is immediately useful because it will detect any extern'd functions or variables that are not being used. It also detects functions or variables that are declared but not defined.

If a FV/VR_EXTERN macro is missing it will be detected either because of a mismatch in the declaration/definition or because a new defined symbol will appear in the nm test.

Eventually the unity build will be used to create a more optimized pgbackrest binary but that will need to wait.
2022-12-31 17:13:41 +07:00
David Steele
8aa2b101bb Put logging functions unused by release builds into DEBUG blocks.
Also inline some functions that are needed in non-DEBUG builds.
2022-12-31 16:41:01 +07:00
David Steele
2332ce8ffc Move storageHelperFree() to storageHelper test harness.
This function was only used for testing so move into a test harness.
2022-12-31 13:14:27 +07:00
David Steele
8b218158ae Move regExpMatchPtr()/regExpMatchStr() to build/common/regExp module.
Similar to b9be4fa5, these functions are not used by the core code so move them to the build module. The new implementation is a little less efficient but that is much less of a worry in the build/test code.

Also remove regExpMatchSize() since it was not longer needed.
2022-12-31 12:54:33 +07:00
David Steele
fa9d831f9f Move xmlNodeAttribute() to build/common/xml module.
Similar to b9be4fa5, this function was not used by the core code so move it to the build module.
2022-12-31 11:09:50 +07:00
David Steele
163a004f30 Move strReplace() and strUpper() to build/common/string module.
Neither of these functions were used by the core code. strReplace() is only used in the tests but it doesn't hurt to put it in build since the build code is not distributed.
2022-12-31 10:26:11 +07:00
David Steele
d517d4a328 Add explicit keyword for covered modules included in other modules.
This was done by checking the extension but it is possible to include a module that does not have a vendor or auto extension. Instead make it explicit that the module is included in another module.

Also change the variable from "include" to "included" to make it clearer what it indicates.
2022-12-31 10:10:44 +07:00
David Steele
cebbf0d012 Remove unused functions.
These functions were either added with the intention that they would be used or they became obsolete over time.
2022-12-30 16:26:48 +07:00
Stefan Fercot
b9be4fa540
Restore errors when no backup matches the current version of PostgreSQL.
It is probably not a good idea to restore the latest backup when it was not made from the current PostgreSQL version. If there is no backup after a stanza-upgrade then replicas might be built with a prior version leading to failures.

Add an error in this case if the latest backup would be used, i.e. --set or --type=time/lsn is not specified.
2022-12-29 15:37:27 +07:00
David Steele
44da314adb
Add compress-level range checking for each compress-type.
The prior range checking was done based on the valid values for gz. While this worked it was a subset of what is available for lz4 and zst.

Allow the range to be specified for each compress-type. Adding this functionality to the parse module would be a better solution but that is a bigger project than this fix deserves, at least for now.
2022-12-27 20:05:08 +07:00
David Steele
56b55f81e8
Add repository checksum to make verify and resume more efficient.
Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data.

This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum.

Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
David Steele
2ab845e263
Store manifest checksums in memory more efficiently.
Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions.

There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
David Steele
77c721eb63
Remove support for PostgreSQL 9.0/9.1/9.2.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.

Remove all logic associated with 9.0/9.1/9.2 and update the tests.

Document the new support policy.

Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
David Steele
010efffb0c Add hex encode/decoding to decode module.
This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
David Steele
9a9ee8e640 Rename EncodeType enum to EncodingType.
This avoids constructs such as decodeToBin(encodeBase64, ...) which are confusing since decode and encode are used in the same function call. decodeToBin(encodingBase64, ...) makes it clearer what is happening.
2022-12-11 18:55:25 +07:00
David Steele
1345caa3f7 Update missing flag in storage/posix test.
The storageNewItrP() permissions test was running twice with the errorOnMissing flag set to false. Fix by setting to true for one test.

Also update the comments to be clearer about what the tests are doing and fix minor formatting.
2022-12-07 20:30:38 +07:00
David Steele
4dc632d570 Add backup test harness.
This allows test backups to be run in other test modules.

It is likely that more logic will be moved here but for now this suffices to get test backups working in the restore module.
2022-12-05 14:15:15 +08:00
David Steele
c4bf775099
Fix missing reference in diff/incr backup.
When loading prior manifests without the new reference list, the code failed to add the current backup to the reference list. Since the current backup is never explicitly referenced, building references from the file list was not sufficient to generate a complete list.

The main problem here was a bad test, fixed in 28f6604. This masked the issue and prevented it from being found. Now it is clear in the test that the current label is missing from the reference list.

Fix by adding the current label to the reference list if a reference list is not stored in the manifest.
2022-11-28 16:42:35 +08:00
David Steele
28f660482a Move manifestBackupLabelSet() test in info/manifest unit.
Changing the label of a manifest that already had a label was not a good test and it ended up masking a bug where the current backup label was not being added to the reference list on manifest load, since manifestBackupLabelSet() added the label to the reference list. In fact, manifestBackupLabelSet() should never be called after a manifest load or even after the label has been set.

Add an assertion to prevent manifestBackupLabelSet() being called when the label is already set.

The bug exposed here will be fixed in a subsequent commit.
2022-11-26 10:48:02 +08:00
David Steele
3f363cb3ae
Add hint when an option is specified without an index.
Hopefully this will make it a little clearer to the user what is wrong when they specify an indexed option without an index.

Also fix an ambiguous use of cfgParseOptionP(). The prior code worked in that it set prefixMatch = true but it was not very readable.
2022-11-22 15:04:13 +08:00
David Steele
70b75532bf v2.42: Bug Fixes
Bug Fixes:

* Fix memory leak in file bundle backup/restore. (Reviewed by John Morris, Oscar. Reported by Oscar.)
* Fix protocol error on short read of remote file. (Reviewed by Stephen Frost.)

Improvements:

* Do not store references for zero-length files when bundling. (Reviewed by Stefan Fercot.)
* Use more generic descriptions for pg_start_backup()/pg_stop_backup(). (Reviewed by Greg Sabino Mullane, David Christensen. Suggested by Greg Sabino Mullane.)

Test Suite Improvements:

* Update test.pl --psql-bin option to match command-line help. (Contributed by Koshi Shibagaki. Reviewed by David Steele.)
2022-11-22 10:20:59 +08:00
k_zshiba
3ad588443b
Update test.pl --psql-bin option to match command-line help.
The option to specify the path to psql was shown in the command-line help as --psql-bin but the option was actually named --pgsql-bin.

Rename to match the help so they are consistent.
2022-11-14 12:47:27 +08:00
David Steele
58b3c91bab Add raw mode to CipherBlock to save space.
The magic in the header is only required so that command-line openssl will recognize the file as being encrypted. In cases where the encrypted data cannot be read with the command-line tool it makes sense to omit the header magic to save some space.

Unfortunately this cannot be enabled for file bundling because it would break backward compatibility. However, it should be possible to enable it for the combination of bundling and block incremental.
2022-11-10 10:28:49 +09:30
David Steele
c9db7bc274 Update cipherBlockNew() to allow optional parameters.
This simplifies calls a bit since digest is never passed and allows for new optional parameters.
2022-11-06 16:12:23 +09:30
David Steele
221db610d2 Shorten names in real/all integration test matrix.
This should allow one or two more parameters to be added without going to a new line, which keeps the matrix easier to read.
2022-10-18 18:02:17 +13:00
David Steele
fee38c2c7c Pass filters to remote storage as a handler array.
The prior code required coverage in the storage/remote module for all filters that could be used remotely.

Now the filter handlers are set at runtime so any filter list can be used with a remote. This is more flexible and makes coverage testing easier. It also resolves a test dependency.

Move the command/remote unit test near the end so it will have access to all filters without using depends.
2022-10-18 16:11:35 +13:00
David Steele
1730ef4ac3
Add noTruncate flag to storageNewWriteP().
This flag skips truncation when opening a file for write on drivers that support it, currently Posix and CIFS. This is convenient for cases where the file needs to be manipulated directly using the file descriptor. Using the file descriptor is not ideal and additional functionality should be added to the storage interface, but for now at least this avoids code duplication, especially on close which updates owners, the timestamp, syncs, etc.

The remote driver forbids no truncate because a file descriptor is never available for a remote storage write object.

Update two instances in the current code which benefit from this new functionality, but the primary reason for the change is to support more complex restore deltas in the upcoming block incremental feature.
2022-10-18 11:33:19 +13:00
David Steele
7967c750d8
Fix protocol error on short read of remote file.
If a remote file read was stopped before the read was complete or if an error occurred in the middle of the read then the protocol would end up in a bad state and produce this error:

ProtocolError: client state is 'data-get' but expected 'idle'

Prevent this by reading the rest of the file on close() or free() to leave the protocol in an idle state for the next command.

This was a possible issue for bundling because the amount to read is known in advance and therefore eof may not be reached. However, I was only able to reproduce this issue with unreleased code.

On error this issue would cause the original error to be lost. The process may still fail with this fix (if the error comes from another source) but hopefully we'll get better information about the original error.
2022-10-18 11:09:00 +13:00
David Steele
ddd966cadc
Use more generic descriptions for pg_start_backup()/pg_stop_backup().
The names were changed in PostgreSQL 15, so update the code and docs to make the naming more generic where needed to avoid using a version-specific name in the logs and documentation.
2022-10-18 09:59:15 +13:00
David Steele
65be4c64a9 Finalize catalog number for PostgreSQL 15 release. 2022-10-16 09:58:35 +13:00
David Steele
909be412f8 Swap command/backup and command/restore unit tests.
Logically restore belongs after backup and in a future commit restore will have a dependency on some backup objects.
2022-10-14 12:08:40 +13:00
David Steele
46a0af3531 Return binary as result from CryptoHash filter.
The prior result was hex-encoded, which is not optimal. This was legacy from the interface with Perl and then the JSON protocol. The new binary protocol natively supports binary so it makes sense to use it and convert to hex where needed.

A number of these hex conversions can now be removed but that will need to be handled in another commit.
2022-10-05 18:05:56 -10:00
David Steele
5602f179a1
Add varint-128 encode/decode to IoRead/IoWrite.
This makes it more efficient to read/write (especially read) varint-128 to/from IO.

Update the Pack type to take advantage of the more efficient read and remove some duplicate code.
2022-10-05 17:01:35 -10:00
David Steele
102ce5dee4
Add persistent reference list to manifest.
The reference list was previously built at load time from whichever references existed in the file list. This was sufficient since the list was for informational purposes only.

The block incremental feature will require a reference list that contains all prior backups, even those that are not explicitly referenced from the manifest. Therefore it makes sense to build and persist a manifest list rather than building it at load time.

This list can still be used for informational purposes, though it needs to be sorted since the list it sill built for older manifest versions and may not be in sorted order.

Add strLstFindIdx() to find references in the list.
2022-10-05 16:28:31 -10:00
David Steele
c647bcb509
Add manifest flags for file processing during backup.
The prior method was to check a combination of fields to determine if a file needed to be copied, delta'd, or resumed. This was complicated and ultimately imposed a limitation on the number of operations that could be performed.

Introduce copy, delta, and resume flags in the manifest to make it clearer which operations need to be performed and to reduce complex and duplicated logic.

This also allows zero-length bundled files to be completed during manifest build rather than later on during backup processing.
2022-10-05 13:14:15 -10:00
David Steele
1ea6a4142e
Improve manifest file updates.
The prior manifestFileUpdate() function was pretty difficult to use since all the parameters had to specified. Instead, pass a ManifestFile struct that has all members set as needed.

When new struct members are added the manifestFileUpdate() call sites will still need to be reviewed, but this should make the process of adding members a bit simpler.
2022-10-04 14:19:12 -10:00
David Steele
f981fb45d9
Do not store references for zero-length files when bundling.
This appears to have been an oversight in 34d6495. Storing the reference is not really correct since the file is not stored in a prior backup. It also uses more space.

There is no real harm in storing the reference, since it is always ignored on restore, but the code is simpler if the zero-length files can be dealt with during the manifest and don't need additional handling later on. This is also an important part of some upcoming optimizations.
2022-10-04 13:22:31 -10:00
David Steele
6e26860c26 Do not log bundle info when a file is delta matched from a prior backup.
It is possible to log the bundle info correctly but the information is useless with the backup reference, which does not appear until later. For now just omit the bundle info so we are not logging something incorrect.
2022-10-02 17:48:43 -10:00
David Steele
4722ad87a7 Add test for differential file bundles to the command/backup unit.
This test exposes a small logging issue. The bundle information for the matched delta on PG_VERSION is not correct. This issue will be fixed in the next commit.

The information stored in the manifest *is* correct so this bug is essentially cosmetic.
2022-10-02 17:41:31 -10:00
David Steele
ac99201c0e Add bufferSize to cvtUInt64FromVarInt128() to further limit reads.
The current call site, manifestFileUnpack(), does not know the total buffer size but the buffer has always been maintained in memory so there should be no corruption. However, there are upcoming use cases where the buffer will be read from IO, the buffer size will be known, and additional sanity checking on buffer overruns will be valuable.

Also rename params to align better with cvtUInt64ToVarInt128().
2022-10-02 17:32:48 -10:00
Reid Thompson
01b81f9d37
Move link creation to storage interface.
Direct link creation via Posix functions has been moved to the Posix driver.

This change allows adding SFTP softlink creation in the SFTP driver using the standard interface.
2022-10-01 15:26:44 -10:00
David Steele
2a4137ed2e Add zero-length chunked content test to common/ioHttp unit.
It seems wasteful to chunk content when there is nothing to send but make sure we handle it gracefully just in case a server decides to do it.
2022-09-25 16:15:23 -08:00
David Steele
64b64b614c Fix comment typo. 2022-09-25 12:30:30 -08:00
David Steele
f1e8e49fa9 Use large error/log buffers in test harness.
Ninja produces quite a bit of output so error messages are often truncated by the default error/log buffers. Use large buffers in the test harness to capture the error even when there is a lot of output.

Ninja has introduced a --quiet option, but it is currently too new to be in any of our test distributions.
2022-09-23 14:00:58 -07:00
David Steele
6b355806cc v2.41: Backup Annotations
Bug Fixes:

* Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
* Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)

Features:

* Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)

Improvements:

* Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
* Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
* Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
* Improve warning message on backup resume. (Suggested by Cynthia Shang.)

Documentation Improvements:

* Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
2022-09-19 10:08:10 -07:00
David Steele
c39c9f220e
Fix issue when listing directories recursively with a filter.
While recursing and filtering, if the last entry in a directory was another directory containing entries then the parent list would get freed too early, causing a double free error or segfault.

Fix by ensuring that the completed list is at the top of the stack before freeing it. This will defer freeing parent lists until the contents of paths have been processed.
2022-09-15 12:00:44 -07:00
David Steele
240cd755d1 Add mem context test missing from 0f7b6a33.
A coverage exception was added during development but was not removed before commit.

Remove the exception and add a test for coverage.
2022-09-14 10:06:06 -07:00
David Steele
8fb61a809d
Add FN_INLINE_ALWAYS macro.
Eliminate a lot of useless repetition for a commonly used pattern.
2022-09-08 18:36:03 -06:00
David Steele
3b5df1e089
Update archive.info timestamps after a successful backup.
Lifecycle policies can cause the archive.info file and its copy to be removed since they are only updated on a stanza-upgrade. Update the timestamps after a successful backup to prevent this.

This does not mean that lifecycle policies should be used as a replacement for expiration. However, in some cases there may be policies in place that are out of admin control. If the lifecycle expiration is less than pgbackrest expiration then corruption of the earliest backup will occur at the very least and there might be other corruption which would make the repo unrecoverable.
2022-09-02 10:03:18 -05:00
David Steele
0f7b6a3344
Skip mem context cleanup in CATCH_FATAL() block.
An error that gets raised all the way to the top TRY block might need to free a lot of resources and any of these callbacks could throw an error and mask the original error. In fact this is pretty likely since we are already in an error state. For example, the Db object will try to close the remote db connection, but if the protocol is in a bad state it will not be able to do so.

Solve this, for now, by not freeing memory or calling callbacks in the CATCH_FATAL() block. This gives us a better chance if being able to report the error without encountering another error first.

For the most part, we don't need to worry about freeing resources (file handles, TLS contexts, etc.) if the program is going to exit immediately. However, it is important to attempt to terminate all active protocol connections, which is done by protocolFree() in main() since the protocol objects live in the top context.

Another way to handle this would be to implement an error stack and that is probably something we will do in the future. But, in the case of a segfault the original error would still be lost. Yet another option would be to still do cleanup but defer it until after the CATCH_FATAL() block.
2022-08-31 12:44:55 -04:00
David Steele
eda7f81ee4
Fix incorrect time expiration being used for non-default repositories.
If a repo is not specified for the expire command then the lowest repo becomes the default. The repo-retention-full value for time was being retrieved from the default rather than a specific repo which led to an incorrect expiration being applied.

Get the value from the specific repo and add a test.

It would be better if the default repo could not be queried in this case but it is not clear how to do that since the repo option is valid for expire (unlike, e.g., archive-push).
2022-08-31 10:01:12 -04:00
David Steele
f1cb848591 Fix comment typos in command/expire unit test. 2022-08-30 18:04:32 -04:00
Stefan Fercot
db75ffd270
Support --set in JSON output for info command.
Allow detailed information (e.g. error list, tablespace list) in JSON output that is already available in text output with the --set option.
2022-08-25 10:12:22 -04:00
Stefan Fercot
381fd0a5a4
Backup key/value annotations.
Allow key/value annotations to be added with the backup command and added/modified/removed with the new annotate command.

Annotations can be viewed with the info command in text mode when --set is specified and are always included in JSON output.
2022-08-24 10:52:33 -04:00
David Steele
b05d31f531
Allow upload chunk size to be configured for object stores.
There are performance benefits to increasing the upload chunk size as long as the tradeoff with additional memory consumption is acceptable.

Make the chunk size configurable for S3, GCS, and Azure, but don't attempt to do any validation of the chunk size beyond some sane limits. The defaults remain as is for each storage type to avoid any unintentional regressions.
2022-08-24 09:33:26 -04:00
David Steele
ff1188f92d Allow quote tag in command-line help.
The quote tag will be used in an upcoming commit.
2022-08-22 13:51:05 -04:00
David Steele
82786da154
Do not allow CATCH() to catch a fatal error.
Catching individual fatal errors was only used in testing so the tests have been updated to use other errors instead. CATCH_FATAL() is now the only way to catch fatal errors.

This simplifies the logic a bit for upcoming changes to error handling and cleanup.

Also fix an issue where passing errorMessage() directly to THROW*() would attempt to copy the message buffer instead of preserving it, which is undefined behavior. Since there were no instances of this behavior before this commit, this was not a live bug.
2022-08-16 16:15:48 -04:00
David Steele
02665a5894 Update Minio test/documentation container version. 2022-08-02 11:05:31 -04:00
David Steele
eb287b18c8 Add profiling, performance, and optimization to C test harness.
All unit and performance tests are now built by the C harness.

Remove all unit/performance test build code from Perl.

Remove code from C harness that is no longer used. This code was included so the C harness could be run separately, but that is no longer needed with this full integration.
2022-07-29 10:31:36 -04:00
David Steele
1e83f2a022 Add coverage to C test harness.
Coverage reports are still generated in Perl, but all the settings have been added to the C harness to generate raw coverage data.
2022-07-28 14:53:48 -04:00
David Steele
c99ea54f17 Integrate C test harness with Perl test harness.
The C test harness is used for unit tests from the Perl harness where possible. Currently, unit tests can be run in the C harness when --no-coverage is specified and --profile is not specified.

C harness tests work on meson 0.45.

The C harness runs with valgrind by default. Valgrind can be disabled with --no-valgrind.

Also rebuild containers to add meson and update the documentation so that meson builds will work (even though we don't do them yet).
2022-07-27 10:32:32 -04:00
David Steele
79d9884141 Move release build check to src/meson.build.
This allows the C harness to perform release builds which are required for some tests.
2022-07-25 09:39:54 -04:00
David Steele
c267ba51b1
Move standby timeline check after checkpoint.
The standby timeline check was being performed using pg_control data loaded before the backup started. If the backup was started immediately after a promotion the standby might not have executed a checkpoint and written the new timeline to pg_control.

Instead perform the timeline check after the checkpoint is executed. This should ensure that the new timeline is in pg_control.
2022-07-22 16:24:55 -04:00
David Steele
cbbe93f592 Improve warning message on backup resume.
The prior warning made it sound as if some action was required on the part of the user.

The new message should make it clearer that this action will be performed by pgBackRest.
2022-07-22 14:45:38 -04:00
David Steele
4c47cc5d47 Remove Debian package patch now that it has been merged upstream. 2022-07-22 09:24:29 -04:00
David Steele
19d9941367 More C test harness improvements and CI.
Build pgbackrest binary and auto-generated code automatically.

Remove --module option and allow modules to run by parameter. This is less verbose and multiple modules can be run at a time.

Allow filtering of modules. Multiple tests can be passed as parameters and if the module ends in / it will be used as a prefix filter. For example, common/ will run all the common modules.

If a test errors the remaining tests will still run but the test process will eventually exit with an error.

CI tests are included but unit tests remain on the development branch.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off) or containerReq (e.g. protocol/protocol) keywords.

Building the C test harness has been simplified:

meson -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug test/build/none pgbackrest
ninja -C test/build/none test/src/test-pgbackrest

To run all modules:

test/build/none/test/src/test-pgbackrest test

Just the common/error module:

test/build/none/test/src/test-pgbackrest test common/error

All info modules:

test/build/none/test/src/test-pgbackrest test info/
2022-07-21 20:10:51 -04:00
David Steele
edfcf1652c Test Dockerfile improvements.
Add tzdata package so timezone tests in command/restore work correctly.

Mark default git path as safe. This is a security fix that is not applicable in this environment, but must be set.

Also remove package cleanup, which is inconvenient when new packages need to be installed. It makes sense for containers that will be downloaded from Dockerhub but not so much for a locally-maintained container.
2022-07-21 19:27:58 -04:00
David Steele
5e5b04be37 Fix common/lock test creating "750" path.
This was clearly an attempt to set the mode when creating a directory, but it never worked and instead created a "750" directory in the current working directory.

Detected when running in an environment where the current working directory was read-only.
2022-07-21 18:19:47 -04:00
David Steele
f9bbafbf3f C test harness improvements.
Add harness depends when present.

Include libyaml in all test builds.

Fix mode on paths before trying to remove and set test path with mode 770 to match the Perl test harness.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off), binReq (e.g. command/archive-get), or containerReq (e.g. protocol/protocol) keywords.

Builds and code generation need to be done in advance. The following commands are required for setup:

meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug build pgbackrest
ninja -C build test/src/test-pgbackrest
build/src/build-code help pgbackrest
build/src/build-code postgres pgbackrest

Now tests can be run, e.g.:

build/test/src/test-pgbackrest --module=postgres/interface
2022-07-20 19:02:14 -04:00
David Steele
c625f05a13 Unify code builder binaries into a single binary.
Creating new binaries was convenient at first but has now become a maintenance issue.

Solve this by combining that into a single binary that takes an additional parameter to indicate which code should be built.

Also clean up path handling to make it easier to build code from the command line.
2022-07-20 17:45:39 -04:00
David Steele
7eb5d67985 Add module harness and shim support to the C test harness.
This allows running unit tests up to common/ini, excluding common/debug-off and common/assert-off.
2022-07-19 18:03:39 -04:00
David Steele
8fdeed7807 Copy repository links as files for testing.
This makes the test code a bit simpler where we are listing a path but not following links.

Links in the repository can be used for testing but should never be committed to the main branch.
2022-07-19 17:57:13 -04:00
David Steele
eb77579008 Fix comment typo. 2022-07-19 17:25:07 -04:00
David Steele
1ff531090b v2.40: OpenSSL 3 Support
NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.

Improvements:

* OpenSSL 3 support. (Reviewed by Stephen Frost.)
* Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)
* Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)
* Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)
* Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris, Soulou. Suggested by Christian Montagne.)

Documentation Bug Fixes:

* Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete.)

Documentation Improvements:

* Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)

Test Suite Improvements:

* Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)
* Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
* Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
2022-07-18 09:32:30 -04:00
David Steele
364af1635d
Force target-timeline=current when restore type=immediate.
Explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository from this backup.

This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at least until they aren't really seen in the wild any longer.

PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly.
2022-07-14 08:26:03 -04:00
David Steele
75623d4583
Create snapshot when listing contents of a path.
Previously a callback was used to list path contents and if no sort was specified then a snapshot was not required. When deleting files from the path some filesystems could omit files that still existed, which meant the path could not be removed.

Filter . out of lists in the Posix driver since this special entry was only used by test code (and filtered everywhere in the core code).

Also remove callbacks from the storage interface and replace with an iterator that should be easier to use and guarantees efficient use of the snapshots.
2022-07-08 17:21:39 -04:00
David Steele
0eccbc8bf4 Meson builds work on version 0.45.
v0.45 ships with Ubuntu 18.04, which is currently the oldest distro we support. We may never do a Meson release on Ubuntu 18.04 but this allows us to start running unit tests with Meson in the meantime.

Some more granular options are not available so we use buildtype in more places.

The check for a in-tree autoconf/make build had to be removed since the filesystem APIs are not available.

Finally, alias_target was removed. This means that full paths must be used for build targets, which does not seem too bad. For instance, test/src/test-pgbackrest must now be used as a build target instead of simple test-pgbackrest.
2022-07-06 18:17:52 -04:00
David Steele
72960bbf17 Rename strTrunc() to strTruncIdx() and add strTrunc().
strTrunc() is now shorthand for truncating to index 0. This is convenient when a string is being reused.
2022-07-06 09:55:32 -04:00
David Steele
1c0bf0b15d Reorder null user/group checks in remote protocol and add tests.
Coverage for these checks was dependent on the order the files were read from disk, which made the tests fragile.

Rearrange the checks and add a test that won't depend on order.
2022-07-05 14:28:40 -04:00
David Steele
845d82f682
Use S3 IsTruncated flag to determine list continuation.
Previously we were just checking for the existence of NextContinuationToken, which the S3 documentation indicates will not be present when the list is not truncated. However, recent versions of Scality send a blank NextContinuationToken when IsTruncated is false. Sending the blank continuation token back causes Scality to send another blank continuation token and an infinite loop occurs.

Instead use IsTruncated (which is required to be present) to determine whether NextContinuationToken should be present. Error if NextContinuationToken is then missing or empty, since an empty token caused an infinite loop with the Scality server (which arguably should have errored when passed an empty token).
2022-07-01 20:32:10 -04:00
David Steele
61ca9b58a0 Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.
The TEST_STORAGE_LIST() macro is more robust and hides the callback mechanism from the caller.

Add features to TEST_STORAGE_LIST() that hrnStorageInfoListCallback() had.

Update tests to use the abbreviated type output (e.g. path/) generated by TEST_STORAGE_LIST().
2022-06-28 14:15:08 -04:00
David Steele
f863fc9888 Add experimental unit test harness written in C.
Having the test harness in C will allow us to remove duplicated Perl code and test on systems where Perl support is not present.

Custom harnesses and shims are currently not implemented, which means only the following tests in the common module will run: error, stack-trace, type-convert, assert-on, mem-context, time, encode, type-object, type-string, type-list, type-buffer, type-variant, reg-exp, log.

The experimental test harness is being committed with partial functionality so it can be used in Windows development. The remaining features will follow as needed.
2022-06-23 12:20:56 -04:00
David Steele
9fd85a104a Disable meson for all but debug builds.
The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.
2022-06-21 09:50:38 -04:00
David Steele
5ecae90f02 Use constants rather than replacements when possible in test.c.
Some of the replacements that were being done already existed as constants, so use the constants instead.

Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().
2022-06-17 16:31:48 -04:00
David Steele
dab1e4b6c6 Add strReplace().
Allows substrings to be replaced with another string.
2022-06-17 16:27:31 -04:00
David Steele
fb9acc1c93 Add higher level YAML functions.
These functions make parsing YAML simpler.
2022-06-17 16:17:02 -04:00
David Steele
55bcb933ee Move protocol module from common to command.
This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.

Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.
2022-06-17 11:17:52 -04:00
Stefan Fercot
eb72a80b47 Fix continuation character spacing. 2022-06-15 09:06:25 -04:00
David Steele
04f5ef25fa Add valgrind to Dockerfile missed in a16cf5ea. 2022-06-09 17:18:32 -04:00
David Steele
79443bea36 Move bldEnum() to the build/common/render module.
This function will be useful for other build/test modules so extern it.

Also skip the first upper-case when no prefix is provided.
2022-06-09 16:55:07 -04:00
David Steele
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
David Steele
08242ee6ac
OpenSSL 3 support.
There are two changes:

* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.

* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.
2022-06-06 14:47:47 -04:00
David Steele
f92ce674f7
Automatically create PostgreSQL version interfaces.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.

Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.

Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.

It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
2022-06-06 13:52:56 -04:00
David Steele
b8fc20d5b8
Add experimental Meson build.
Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.

The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.

Also patch the Debian package to force autoconf/make rather than Meson.
2022-06-03 14:13:56 -04:00
David Steele
148956aed8 Remove useless command/check unit test.
This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.
2022-06-01 10:13:57 -04:00
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
Reid Thompson
60d70fa66f
Disable incremental manifest save when resume=n.
The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.
2022-05-25 18:18:37 -04:00
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00
David Steele
4dcc9df222 Add labels in parse.auto.c to make diffs easier to read.
Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.

This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.

The change to parse.auto.c will be committed separately so it can be ignored in history/blame.
2022-05-16 09:14:46 -04:00
David Steele
901e829f6d v2.39: Verify and File Bundling
Bug Fixes:

* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)

Features:

* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)

Improvements:

* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)

Documentation Bug Fixes:

* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)

Documentation Improvements:

* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 08:46:24 -04:00
David Steele
19dd015d58
Fix issues in improved path validation for repo-* commands.
If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:

$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
ASSERT: [025]: start <= this->pub.size (on dev builds)
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)

Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.

Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.

This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.
2022-05-13 09:41:53 -04:00
David Steele
024500782e Reduce encrypted WAL segments sizes in command tests.
The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.

Reduce the size of the encrypted test segments where possible to improve overall test performance.
2022-05-12 17:02:08 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
Reid Thompson
a913113fda
Add option type descriptions.
This cuts down on repetition of the size descriptions and adds basic descriptions for the other option types.
2022-05-11 15:23:41 -04:00
David Steele
8ee85bc605 Fix "that that" typos. 2022-05-11 08:46:23 -04:00
David Steele
de816a0f57
Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
David Steele
3a40394472 Remove obsolete test in common/memContext.
Once upon a time the allocation array was allocated up front so this test was required for the top context, which did not allocate up front.

Now allocations are done on demand so this case is covered for every context that does not allocate memory.
2022-05-10 11:18:12 -04:00
Reid Thompson
6b98b3534e Fix typo. 2022-05-10 06:52:56 -04:00
David Steele
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
David Steele
39dddbb6bc Add limited CI for ppc64le/s390x using emulation.
The emulation is so slow that running all the unit tests would be too expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to test on one big-endian architecture to be sure that checksums are correct.

Update checksums in the tests where they had gotten out of date since the last time we were testing on s390x. Also use a different test in command/archivePushTest to show the name of the file when a checksum does not match to aid in debugging.

The command/archive-push test was updated but not included because there is also a permissions issue, which looks to be the same as what we see on MacOS/FreedBSD. Hopefully we'll be able to fix all of those at the same time.
2022-05-09 12:48:19 -04:00
David Steele
ef4c4ab852
Use variable instead of function to track FINALLY() state.
The function worked fine, but Coverity was unable to determine that the finally block was run, which led to false positives about unfreed memory.

Using a boolean in the block makes it clear to Coverity that the finally block will always be run no matter what else happens.

We'll depend on the compiler to optimize away the boolean if it is not used in a finally block. The cost of the boolean is fairly low in comparison to everything else being done in these macros, so it does not seem worth having a separate block even if the compiler is not able to eliminate the boolean.

This reverts most of 9a271e9 that fixed a bug caused by c5b5b58, which was also attempting to help Coverity understand FINALLY() blocks.
2022-05-09 10:39:43 -04:00
David Steele
46b7b72874 Add hint when unable to find the primary cluster.
If all available clusters are in recovery, pgBackRest will not be able to find a primary for the backup.
2022-05-06 18:23:36 -04:00
David Steele
53bfdbc01e Remove useless test in config/parse unit test.
Since the packSize field is 7 bits, it could never fail the check for > 127.

The compiler will catch any packs that are larger than 7 bits and then the pack size will need to be adjusted. For now just adjust the comment to reflect what the test does and give a clearer indication of what to do when a pack grows too large.
2022-05-06 16:02:44 -04:00
David Steele
77311a9af7 Fix indentation.
gcc11 complains about this indentation being misleading.
2022-05-06 15:38:03 -04:00
David Steele
efbcd975c4 Update Docker test image for Debian 9.
A change invalidated the current image which has been causing the d9 test to run longer.
2022-05-06 13:56:58 -04:00
David Steele
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
Reid Thompson
65d22e4325
Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
David Steele
e70c71049e Use uint8_t for optionResolveOrder.
This saves a bit of space and should not affect processing speed.

On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.
2022-05-06 07:49:23 -04:00
David Steele
808f7bf11c Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.
This test was likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:14:13 -04:00
David Steele
efe0a39a75 Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.
These tests were likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:01:02 -04:00