1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2381 Commits

Author SHA1 Message Date
David Steele
16ac5ee8d3 Rename block incremental manifest keys.
Since the keys need to be read/written in order, these keys make the logic a bit simpler.
2023-02-26 16:13:44 +07:00
David Steele
a9867cb0b8 Add repo-block-age-map and repo-block-size-map options.
Make these options configurable. This is primarily for testing purposes so the new options will be kept internal.
2023-02-26 14:49:34 +07:00
David Steele
dffc933384 Rename DeltaMap to BlockHash.
This more accurately describes what the object does.
2023-02-13 09:17:30 +07:00
David Steele
779efe0d7a Consistently declare block incremental size as size_t.
The block is often allocated in memory so size_t makes more sense than uint64_t.
2023-02-09 13:01:56 +07:00
David Steele
089fae035b Add block incremental to real/all test output. 2023-02-07 14:09:50 +07:00
David Steele
31cad5e09e Check for stray execute permissions in test.pl --code-format.
Sometimes these can get set while updating permissions from with a VM or container.

Also fix a few permissions that were not correct.
2023-02-01 12:57:04 +07:00
David Steele
c5907a2e71 Remove references to SSH made obsolete when TLS was introduced.
Also remove details about SSH compression that are not helpful.
2023-01-31 08:28:32 +07:00
David Steele
ce0ea2cfab Use uncrustify for code formatting.
The code is not completely reflowed yet so there are some cases that uncrustify will not catch. The formatting will be improved over time.

Some block of code require special formatting so have been surrounded with the {uncrustify-off}/{uncrustify-on} markers. These exceptions should be kept to a minimum.

Add --code-format (to reformat code) and --code-format-check (to check formatting) to test.pl.

Add a CI test that will check code formatting. Code must be correctly formatted before it can be merge to integration.

Add documentation to the coding standards for code formatting.
2023-01-30 12:22:41 +07:00
David Steele
d4070c9064 Reformat code with uncrustify.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:

* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.

Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.

Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
David Steele
b2202c36d9 Fix formatting errors.
Errors in our current (manually-maintained) code format discovered by uncrustify.
2023-01-30 11:16:31 +07:00
David Steele
f91af30584 Add FN_PRINTF() macro.
Bring the format(printf) attribute in line with the FN_NO_RETURN and FN_INLINE_ALWAYS macros.

This is simpler to read and can be customized for different compilers.
2023-01-30 10:51:48 +07:00
David Steele
f8ea51bf8f Refactor stackTraceToZ() to avoid multiple versions of the function.
stackTraceToZ() was split this way in c8264291 to allow complete coverage. 0becb6da added a shim to improve coveage but missed simplifying the function.
2023-01-30 10:27:17 +07:00
David Steele
053468bfb1 v2.44: Remove PostgreSQL 9.0/9.1/9.2 Support
Improvements:

* Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
* Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
* Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)

Documentation Improvements:

* Add warning about enabling "hierarchical namespace" on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
* Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
* Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
* Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
* Add dark mode to the website. (Suggested by Stephen Frost.)
2023-01-30 09:15:44 +07:00
David Steele
a28f3d49c2 Add --no-back-trace option to test.pl.
Running valgrind and backtrace together has been causing tests to timeout in CI, mostly likely due to limited resources. This has not been a problem in normal development environments.

Since it is still important to run backtraces for debugging, split the u22 test that was doing all this work to run coverage and backtrace together and valgrind-only as a separate test. As a bonus these tests run faster separately and since they run in parallel the total execution time is faster.
2023-01-28 18:55:53 +07:00
David Steele
16c625353d Skip test modules in coverage report.
Coverage of the test code is useful during development but it does not seem worth adding it to the core code coverage report.
2023-01-28 16:22:04 +07:00
David Steele
d5ef4ce747 Fix permissions on source files. 2023-01-28 11:01:29 +07:00
David Steele
1da2666a9e Add manifest test harness.
These macros make adding paths/files/etc to a manifest simpler and easier to read.
2023-01-21 14:03:27 +07:00
David Steele
912eec63bb
Block-level incremental backup.
The primary goal of the block incremental backup is to save space in the repository by only storing changed parts of a file rather than the entire file. This implementation is focused on restore performance more than saving space in the repository, though there may be substantial savings depending on the workload.

The repo-block option enables the feature (when repo-bundle is already enabled). The block size is determined based on the file size and age. Very old or very small files will not use block incremental.
2023-01-20 16:48:57 +07:00
David Steele
008a18555c Fix comment indentation and wrapping. 2023-01-20 15:44:01 +07:00
David Steele
bfc9a5c33e Fix incorrect FUNCTION_TEST_END() in hrnPgWalToBuffer(). 2023-01-14 18:01:48 +07:00
David Steele
34e4835ff3
Refactor common/ini module to remove callbacks and duplicated code.
The callbacks in iniLoad() made the downstream code more complicated than it needed to be so use an iterator model instead.

Combine the two functions that were used to load the ini data to remove code duplication. In theory it would be nice to use iniValueNext() in the config/parse module rather than loading a KeyValue store but this would mean a big change to the parser, which does not seem worthwhile at this time.
2023-01-12 21:24:28 +07:00
David Steele
9ca492cecf Audit mem contexts returned from functions into the calling context.
It is possible for functions to accidentally leak child contexts into the calling context, which may use a lot of memory depending on the use case and where it happens.

Use the function return type to determine what should be returned and error when something else is returned. Add FUNCTION_AUDIT_*() macros to handle exceptions.

This checking is only performed during unit tests on the code being covered by the specific unit test.

Note that this does not work yet for memory allocations, i.e. memNew(). These are pretty rare so are not as much of an issue and they can be added in the future.
2023-01-12 17:36:57 +07:00
David Steele
de1dfb66ca Refactor logging functions to never allocate memory.
Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case.

This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit.

Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
David Steele
0becb6da31 Enhance libbacktrace to handle incomplete stack traces.
This fills in backtrace info at the bottom of the call stack when the stack trace is incomplete due to testing. This does not affect release builds, which is why it did not make the first cut, but it turns out to be useful for testing and barely changes the release code (when we do release this).

The recursion test in common/error was simplified because it would now return a very large trace.
2023-01-12 10:22:26 +07:00
David Steele
9347313cbf Fix error detail output to stderr.
The error detail should be output when the error is an assert (this part was working) or the log level is at least debug. In cases where log-level-console was at least debug but log-level-stderr was not the detail was lost.

Improve the range checking to output error detail to stderr when log-level-console is at least debug.
2023-01-11 19:16:31 +07:00
David Steele
c826429123 Improve libbacktrace implementation.
The libbacktrace feature has not been working since the move to meson because libbacktrace detection was not added to the meson build. Add libbacktrace to meson and improve the feature so that it can be compiled into release builds.

The prior implementation fetched line numbers with each stack trace push. Not only was this slow but it missed any functions that were not being tracked on our stack.

Instead just examine the backtrace when an error happens and merge it with the info we have on our stack. If the backtrace is not available then the output remains as before.

Also remove --backtrace from test.pl since the library is now auto-detected.

Leave this library out of the production build for now to give it a little time to shake out in testing.
2023-01-11 11:19:26 +07:00
David Steele
6633ccd18d Do not output stderr on unit/integration test errors.
Since stderr is being redirected to stdout this results in duplication of the error output.
2023-01-05 14:03:43 +07:00
David Steele
4429bc82f5 Add unit tests for the unit test build code.
When this code was migrated to C the unit tests were not included because there were more important priorities at the time.

This also requires some adjustments to coverage because of the new code location.
2023-01-05 12:59:06 +07:00
David Steele
a5499afabc Add CHECK_FMT() macros and tests for CHECK*() macros.
The CHECK_FMT() macro will be used in some upcoming tests but could also be useful in the core code.

The CHECK*() macros never had tests so add them.
2023-01-05 12:44:00 +07:00
David Steele
d51a86c621 Add BUFFER_EXTERN() and cleanup hash zero constants.
BUFFER_EXTERN() provides a clean way to create buffer constants.

Convert HASH_TYPE_SHA256_ZERO_STR to HASH_TYPE_SHA256_ZERO_BUF to be consistent with HASH_TYPE_SHA1_ZERO_BUF.
2023-01-02 17:30:57 +07:00
David Steele
f018912908 Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN.
This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can).

Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
David Steele
4fb8a0ecdd Add meson unity build and tests.
This is immediately useful because it will detect any extern'd functions or variables that are not being used. It also detects functions or variables that are declared but not defined.

If a FV/VR_EXTERN macro is missing it will be detected either because of a mismatch in the declaration/definition or because a new defined symbol will appear in the nm test.

Eventually the unity build will be used to create a more optimized pgbackrest binary but that will need to wait.
2022-12-31 17:13:41 +07:00
David Steele
8aa2b101bb Put logging functions unused by release builds into DEBUG blocks.
Also inline some functions that are needed in non-DEBUG builds.
2022-12-31 16:41:01 +07:00
David Steele
2332ce8ffc Move storageHelperFree() to storageHelper test harness.
This function was only used for testing so move into a test harness.
2022-12-31 13:14:27 +07:00
David Steele
8b218158ae Move regExpMatchPtr()/regExpMatchStr() to build/common/regExp module.
Similar to b9be4fa5, these functions are not used by the core code so move them to the build module. The new implementation is a little less efficient but that is much less of a worry in the build/test code.

Also remove regExpMatchSize() since it was not longer needed.
2022-12-31 12:54:33 +07:00
David Steele
fa9d831f9f Move xmlNodeAttribute() to build/common/xml module.
Similar to b9be4fa5, this function was not used by the core code so move it to the build module.
2022-12-31 11:09:50 +07:00
David Steele
163a004f30 Move strReplace() and strUpper() to build/common/string module.
Neither of these functions were used by the core code. strReplace() is only used in the tests but it doesn't hurt to put it in build since the build code is not distributed.
2022-12-31 10:26:11 +07:00
David Steele
d517d4a328 Add explicit keyword for covered modules included in other modules.
This was done by checking the extension but it is possible to include a module that does not have a vendor or auto extension. Instead make it explicit that the module is included in another module.

Also change the variable from "include" to "included" to make it clearer what it indicates.
2022-12-31 10:10:44 +07:00
David Steele
cebbf0d012 Remove unused functions.
These functions were either added with the intention that they would be used or they became obsolete over time.
2022-12-30 16:26:48 +07:00
Stefan Fercot
b9be4fa540
Restore errors when no backup matches the current version of PostgreSQL.
It is probably not a good idea to restore the latest backup when it was not made from the current PostgreSQL version. If there is no backup after a stanza-upgrade then replicas might be built with a prior version leading to failures.

Add an error in this case if the latest backup would be used, i.e. --set or --type=time/lsn is not specified.
2022-12-29 15:37:27 +07:00
David Steele
44da314adb
Add compress-level range checking for each compress-type.
The prior range checking was done based on the valid values for gz. While this worked it was a subset of what is available for lz4 and zst.

Allow the range to be specified for each compress-type. Adding this functionality to the parse module would be a better solution but that is a bigger project than this fix deserves, at least for now.
2022-12-27 20:05:08 +07:00
David Steele
56b55f81e8
Add repository checksum to make verify and resume more efficient.
Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data.

This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum.

Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
David Steele
2ab845e263
Store manifest checksums in memory more efficiently.
Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions.

There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
David Steele
77c721eb63
Remove support for PostgreSQL 9.0/9.1/9.2.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.

Remove all logic associated with 9.0/9.1/9.2 and update the tests.

Document the new support policy.

Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
David Steele
010efffb0c Add hex encode/decoding to decode module.
This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
David Steele
9a9ee8e640 Rename EncodeType enum to EncodingType.
This avoids constructs such as decodeToBin(encodeBase64, ...) which are confusing since decode and encode are used in the same function call. decodeToBin(encodingBase64, ...) makes it clearer what is happening.
2022-12-11 18:55:25 +07:00
David Steele
1345caa3f7 Update missing flag in storage/posix test.
The storageNewItrP() permissions test was running twice with the errorOnMissing flag set to false. Fix by setting to true for one test.

Also update the comments to be clearer about what the tests are doing and fix minor formatting.
2022-12-07 20:30:38 +07:00
David Steele
4dc632d570 Add backup test harness.
This allows test backups to be run in other test modules.

It is likely that more logic will be moved here but for now this suffices to get test backups working in the restore module.
2022-12-05 14:15:15 +08:00
David Steele
c4bf775099
Fix missing reference in diff/incr backup.
When loading prior manifests without the new reference list, the code failed to add the current backup to the reference list. Since the current backup is never explicitly referenced, building references from the file list was not sufficient to generate a complete list.

The main problem here was a bad test, fixed in 28f6604. This masked the issue and prevented it from being found. Now it is clear in the test that the current label is missing from the reference list.

Fix by adding the current label to the reference list if a reference list is not stored in the manifest.
2022-11-28 16:42:35 +08:00
David Steele
28f660482a Move manifestBackupLabelSet() test in info/manifest unit.
Changing the label of a manifest that already had a label was not a good test and it ended up masking a bug where the current backup label was not being added to the reference list on manifest load, since manifestBackupLabelSet() added the label to the reference list. In fact, manifestBackupLabelSet() should never be called after a manifest load or even after the label has been set.

Add an assertion to prevent manifestBackupLabelSet() being called when the label is already set.

The bug exposed here will be fixed in a subsequent commit.
2022-11-26 10:48:02 +08:00
David Steele
3f363cb3ae
Add hint when an option is specified without an index.
Hopefully this will make it a little clearer to the user what is wrong when they specify an indexed option without an index.

Also fix an ambiguous use of cfgParseOptionP(). The prior code worked in that it set prefixMatch = true but it was not very readable.
2022-11-22 15:04:13 +08:00
David Steele
70b75532bf v2.42: Bug Fixes
Bug Fixes:

* Fix memory leak in file bundle backup/restore. (Reviewed by John Morris, Oscar. Reported by Oscar.)
* Fix protocol error on short read of remote file. (Reviewed by Stephen Frost.)

Improvements:

* Do not store references for zero-length files when bundling. (Reviewed by Stefan Fercot.)
* Use more generic descriptions for pg_start_backup()/pg_stop_backup(). (Reviewed by Greg Sabino Mullane, David Christensen. Suggested by Greg Sabino Mullane.)

Test Suite Improvements:

* Update test.pl --psql-bin option to match command-line help. (Contributed by Koshi Shibagaki. Reviewed by David Steele.)
2022-11-22 10:20:59 +08:00
k_zshiba
3ad588443b
Update test.pl --psql-bin option to match command-line help.
The option to specify the path to psql was shown in the command-line help as --psql-bin but the option was actually named --pgsql-bin.

Rename to match the help so they are consistent.
2022-11-14 12:47:27 +08:00
David Steele
58b3c91bab Add raw mode to CipherBlock to save space.
The magic in the header is only required so that command-line openssl will recognize the file as being encrypted. In cases where the encrypted data cannot be read with the command-line tool it makes sense to omit the header magic to save some space.

Unfortunately this cannot be enabled for file bundling because it would break backward compatibility. However, it should be possible to enable it for the combination of bundling and block incremental.
2022-11-10 10:28:49 +09:30
David Steele
c9db7bc274 Update cipherBlockNew() to allow optional parameters.
This simplifies calls a bit since digest is never passed and allows for new optional parameters.
2022-11-06 16:12:23 +09:30
David Steele
221db610d2 Shorten names in real/all integration test matrix.
This should allow one or two more parameters to be added without going to a new line, which keeps the matrix easier to read.
2022-10-18 18:02:17 +13:00
David Steele
fee38c2c7c Pass filters to remote storage as a handler array.
The prior code required coverage in the storage/remote module for all filters that could be used remotely.

Now the filter handlers are set at runtime so any filter list can be used with a remote. This is more flexible and makes coverage testing easier. It also resolves a test dependency.

Move the command/remote unit test near the end so it will have access to all filters without using depends.
2022-10-18 16:11:35 +13:00
David Steele
1730ef4ac3
Add noTruncate flag to storageNewWriteP().
This flag skips truncation when opening a file for write on drivers that support it, currently Posix and CIFS. This is convenient for cases where the file needs to be manipulated directly using the file descriptor. Using the file descriptor is not ideal and additional functionality should be added to the storage interface, but for now at least this avoids code duplication, especially on close which updates owners, the timestamp, syncs, etc.

The remote driver forbids no truncate because a file descriptor is never available for a remote storage write object.

Update two instances in the current code which benefit from this new functionality, but the primary reason for the change is to support more complex restore deltas in the upcoming block incremental feature.
2022-10-18 11:33:19 +13:00
David Steele
7967c750d8
Fix protocol error on short read of remote file.
If a remote file read was stopped before the read was complete or if an error occurred in the middle of the read then the protocol would end up in a bad state and produce this error:

ProtocolError: client state is 'data-get' but expected 'idle'

Prevent this by reading the rest of the file on close() or free() to leave the protocol in an idle state for the next command.

This was a possible issue for bundling because the amount to read is known in advance and therefore eof may not be reached. However, I was only able to reproduce this issue with unreleased code.

On error this issue would cause the original error to be lost. The process may still fail with this fix (if the error comes from another source) but hopefully we'll get better information about the original error.
2022-10-18 11:09:00 +13:00
David Steele
ddd966cadc
Use more generic descriptions for pg_start_backup()/pg_stop_backup().
The names were changed in PostgreSQL 15, so update the code and docs to make the naming more generic where needed to avoid using a version-specific name in the logs and documentation.
2022-10-18 09:59:15 +13:00
David Steele
65be4c64a9 Finalize catalog number for PostgreSQL 15 release. 2022-10-16 09:58:35 +13:00
David Steele
909be412f8 Swap command/backup and command/restore unit tests.
Logically restore belongs after backup and in a future commit restore will have a dependency on some backup objects.
2022-10-14 12:08:40 +13:00
David Steele
46a0af3531 Return binary as result from CryptoHash filter.
The prior result was hex-encoded, which is not optimal. This was legacy from the interface with Perl and then the JSON protocol. The new binary protocol natively supports binary so it makes sense to use it and convert to hex where needed.

A number of these hex conversions can now be removed but that will need to be handled in another commit.
2022-10-05 18:05:56 -10:00
David Steele
5602f179a1
Add varint-128 encode/decode to IoRead/IoWrite.
This makes it more efficient to read/write (especially read) varint-128 to/from IO.

Update the Pack type to take advantage of the more efficient read and remove some duplicate code.
2022-10-05 17:01:35 -10:00
David Steele
102ce5dee4
Add persistent reference list to manifest.
The reference list was previously built at load time from whichever references existed in the file list. This was sufficient since the list was for informational purposes only.

The block incremental feature will require a reference list that contains all prior backups, even those that are not explicitly referenced from the manifest. Therefore it makes sense to build and persist a manifest list rather than building it at load time.

This list can still be used for informational purposes, though it needs to be sorted since the list it sill built for older manifest versions and may not be in sorted order.

Add strLstFindIdx() to find references in the list.
2022-10-05 16:28:31 -10:00
David Steele
c647bcb509
Add manifest flags for file processing during backup.
The prior method was to check a combination of fields to determine if a file needed to be copied, delta'd, or resumed. This was complicated and ultimately imposed a limitation on the number of operations that could be performed.

Introduce copy, delta, and resume flags in the manifest to make it clearer which operations need to be performed and to reduce complex and duplicated logic.

This also allows zero-length bundled files to be completed during manifest build rather than later on during backup processing.
2022-10-05 13:14:15 -10:00
David Steele
1ea6a4142e
Improve manifest file updates.
The prior manifestFileUpdate() function was pretty difficult to use since all the parameters had to specified. Instead, pass a ManifestFile struct that has all members set as needed.

When new struct members are added the manifestFileUpdate() call sites will still need to be reviewed, but this should make the process of adding members a bit simpler.
2022-10-04 14:19:12 -10:00
David Steele
f981fb45d9
Do not store references for zero-length files when bundling.
This appears to have been an oversight in 34d6495. Storing the reference is not really correct since the file is not stored in a prior backup. It also uses more space.

There is no real harm in storing the reference, since it is always ignored on restore, but the code is simpler if the zero-length files can be dealt with during the manifest and don't need additional handling later on. This is also an important part of some upcoming optimizations.
2022-10-04 13:22:31 -10:00
David Steele
6e26860c26 Do not log bundle info when a file is delta matched from a prior backup.
It is possible to log the bundle info correctly but the information is useless with the backup reference, which does not appear until later. For now just omit the bundle info so we are not logging something incorrect.
2022-10-02 17:48:43 -10:00
David Steele
4722ad87a7 Add test for differential file bundles to the command/backup unit.
This test exposes a small logging issue. The bundle information for the matched delta on PG_VERSION is not correct. This issue will be fixed in the next commit.

The information stored in the manifest *is* correct so this bug is essentially cosmetic.
2022-10-02 17:41:31 -10:00
David Steele
ac99201c0e Add bufferSize to cvtUInt64FromVarInt128() to further limit reads.
The current call site, manifestFileUnpack(), does not know the total buffer size but the buffer has always been maintained in memory so there should be no corruption. However, there are upcoming use cases where the buffer will be read from IO, the buffer size will be known, and additional sanity checking on buffer overruns will be valuable.

Also rename params to align better with cvtUInt64ToVarInt128().
2022-10-02 17:32:48 -10:00
Reid Thompson
01b81f9d37
Move link creation to storage interface.
Direct link creation via Posix functions has been moved to the Posix driver.

This change allows adding SFTP softlink creation in the SFTP driver using the standard interface.
2022-10-01 15:26:44 -10:00
David Steele
2a4137ed2e Add zero-length chunked content test to common/ioHttp unit.
It seems wasteful to chunk content when there is nothing to send but make sure we handle it gracefully just in case a server decides to do it.
2022-09-25 16:15:23 -08:00
David Steele
64b64b614c Fix comment typo. 2022-09-25 12:30:30 -08:00
David Steele
f1e8e49fa9 Use large error/log buffers in test harness.
Ninja produces quite a bit of output so error messages are often truncated by the default error/log buffers. Use large buffers in the test harness to capture the error even when there is a lot of output.

Ninja has introduced a --quiet option, but it is currently too new to be in any of our test distributions.
2022-09-23 14:00:58 -07:00
David Steele
6b355806cc v2.41: Backup Annotations
Bug Fixes:

* Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
* Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)

Features:

* Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)

Improvements:

* Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
* Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
* Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
* Improve warning message on backup resume. (Suggested by Cynthia Shang.)

Documentation Improvements:

* Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
2022-09-19 10:08:10 -07:00
David Steele
c39c9f220e
Fix issue when listing directories recursively with a filter.
While recursing and filtering, if the last entry in a directory was another directory containing entries then the parent list would get freed too early, causing a double free error or segfault.

Fix by ensuring that the completed list is at the top of the stack before freeing it. This will defer freeing parent lists until the contents of paths have been processed.
2022-09-15 12:00:44 -07:00
David Steele
240cd755d1 Add mem context test missing from 0f7b6a33.
A coverage exception was added during development but was not removed before commit.

Remove the exception and add a test for coverage.
2022-09-14 10:06:06 -07:00
David Steele
8fb61a809d
Add FN_INLINE_ALWAYS macro.
Eliminate a lot of useless repetition for a commonly used pattern.
2022-09-08 18:36:03 -06:00
David Steele
3b5df1e089
Update archive.info timestamps after a successful backup.
Lifecycle policies can cause the archive.info file and its copy to be removed since they are only updated on a stanza-upgrade. Update the timestamps after a successful backup to prevent this.

This does not mean that lifecycle policies should be used as a replacement for expiration. However, in some cases there may be policies in place that are out of admin control. If the lifecycle expiration is less than pgbackrest expiration then corruption of the earliest backup will occur at the very least and there might be other corruption which would make the repo unrecoverable.
2022-09-02 10:03:18 -05:00
David Steele
0f7b6a3344
Skip mem context cleanup in CATCH_FATAL() block.
An error that gets raised all the way to the top TRY block might need to free a lot of resources and any of these callbacks could throw an error and mask the original error. In fact this is pretty likely since we are already in an error state. For example, the Db object will try to close the remote db connection, but if the protocol is in a bad state it will not be able to do so.

Solve this, for now, by not freeing memory or calling callbacks in the CATCH_FATAL() block. This gives us a better chance if being able to report the error without encountering another error first.

For the most part, we don't need to worry about freeing resources (file handles, TLS contexts, etc.) if the program is going to exit immediately. However, it is important to attempt to terminate all active protocol connections, which is done by protocolFree() in main() since the protocol objects live in the top context.

Another way to handle this would be to implement an error stack and that is probably something we will do in the future. But, in the case of a segfault the original error would still be lost. Yet another option would be to still do cleanup but defer it until after the CATCH_FATAL() block.
2022-08-31 12:44:55 -04:00
David Steele
eda7f81ee4
Fix incorrect time expiration being used for non-default repositories.
If a repo is not specified for the expire command then the lowest repo becomes the default. The repo-retention-full value for time was being retrieved from the default rather than a specific repo which led to an incorrect expiration being applied.

Get the value from the specific repo and add a test.

It would be better if the default repo could not be queried in this case but it is not clear how to do that since the repo option is valid for expire (unlike, e.g., archive-push).
2022-08-31 10:01:12 -04:00
David Steele
f1cb848591 Fix comment typos in command/expire unit test. 2022-08-30 18:04:32 -04:00
Stefan Fercot
db75ffd270
Support --set in JSON output for info command.
Allow detailed information (e.g. error list, tablespace list) in JSON output that is already available in text output with the --set option.
2022-08-25 10:12:22 -04:00
Stefan Fercot
381fd0a5a4
Backup key/value annotations.
Allow key/value annotations to be added with the backup command and added/modified/removed with the new annotate command.

Annotations can be viewed with the info command in text mode when --set is specified and are always included in JSON output.
2022-08-24 10:52:33 -04:00
David Steele
b05d31f531
Allow upload chunk size to be configured for object stores.
There are performance benefits to increasing the upload chunk size as long as the tradeoff with additional memory consumption is acceptable.

Make the chunk size configurable for S3, GCS, and Azure, but don't attempt to do any validation of the chunk size beyond some sane limits. The defaults remain as is for each storage type to avoid any unintentional regressions.
2022-08-24 09:33:26 -04:00
David Steele
ff1188f92d Allow quote tag in command-line help.
The quote tag will be used in an upcoming commit.
2022-08-22 13:51:05 -04:00
David Steele
82786da154
Do not allow CATCH() to catch a fatal error.
Catching individual fatal errors was only used in testing so the tests have been updated to use other errors instead. CATCH_FATAL() is now the only way to catch fatal errors.

This simplifies the logic a bit for upcoming changes to error handling and cleanup.

Also fix an issue where passing errorMessage() directly to THROW*() would attempt to copy the message buffer instead of preserving it, which is undefined behavior. Since there were no instances of this behavior before this commit, this was not a live bug.
2022-08-16 16:15:48 -04:00
David Steele
02665a5894 Update Minio test/documentation container version. 2022-08-02 11:05:31 -04:00
David Steele
eb287b18c8 Add profiling, performance, and optimization to C test harness.
All unit and performance tests are now built by the C harness.

Remove all unit/performance test build code from Perl.

Remove code from C harness that is no longer used. This code was included so the C harness could be run separately, but that is no longer needed with this full integration.
2022-07-29 10:31:36 -04:00
David Steele
1e83f2a022 Add coverage to C test harness.
Coverage reports are still generated in Perl, but all the settings have been added to the C harness to generate raw coverage data.
2022-07-28 14:53:48 -04:00
David Steele
c99ea54f17 Integrate C test harness with Perl test harness.
The C test harness is used for unit tests from the Perl harness where possible. Currently, unit tests can be run in the C harness when --no-coverage is specified and --profile is not specified.

C harness tests work on meson 0.45.

The C harness runs with valgrind by default. Valgrind can be disabled with --no-valgrind.

Also rebuild containers to add meson and update the documentation so that meson builds will work (even though we don't do them yet).
2022-07-27 10:32:32 -04:00
David Steele
79d9884141 Move release build check to src/meson.build.
This allows the C harness to perform release builds which are required for some tests.
2022-07-25 09:39:54 -04:00
David Steele
c267ba51b1
Move standby timeline check after checkpoint.
The standby timeline check was being performed using pg_control data loaded before the backup started. If the backup was started immediately after a promotion the standby might not have executed a checkpoint and written the new timeline to pg_control.

Instead perform the timeline check after the checkpoint is executed. This should ensure that the new timeline is in pg_control.
2022-07-22 16:24:55 -04:00
David Steele
cbbe93f592 Improve warning message on backup resume.
The prior warning made it sound as if some action was required on the part of the user.

The new message should make it clearer that this action will be performed by pgBackRest.
2022-07-22 14:45:38 -04:00
David Steele
4c47cc5d47 Remove Debian package patch now that it has been merged upstream. 2022-07-22 09:24:29 -04:00
David Steele
19d9941367 More C test harness improvements and CI.
Build pgbackrest binary and auto-generated code automatically.

Remove --module option and allow modules to run by parameter. This is less verbose and multiple modules can be run at a time.

Allow filtering of modules. Multiple tests can be passed as parameters and if the module ends in / it will be used as a prefix filter. For example, common/ will run all the common modules.

If a test errors the remaining tests will still run but the test process will eventually exit with an error.

CI tests are included but unit tests remain on the development branch.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off) or containerReq (e.g. protocol/protocol) keywords.

Building the C test harness has been simplified:

meson -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug test/build/none pgbackrest
ninja -C test/build/none test/src/test-pgbackrest

To run all modules:

test/build/none/test/src/test-pgbackrest test

Just the common/error module:

test/build/none/test/src/test-pgbackrest test common/error

All info modules:

test/build/none/test/src/test-pgbackrest test info/
2022-07-21 20:10:51 -04:00
David Steele
edfcf1652c Test Dockerfile improvements.
Add tzdata package so timezone tests in command/restore work correctly.

Mark default git path as safe. This is a security fix that is not applicable in this environment, but must be set.

Also remove package cleanup, which is inconvenient when new packages need to be installed. It makes sense for containers that will be downloaded from Dockerhub but not so much for a locally-maintained container.
2022-07-21 19:27:58 -04:00
David Steele
5e5b04be37 Fix common/lock test creating "750" path.
This was clearly an attempt to set the mode when creating a directory, but it never worked and instead created a "750" directory in the current working directory.

Detected when running in an environment where the current working directory was read-only.
2022-07-21 18:19:47 -04:00
David Steele
f9bbafbf3f C test harness improvements.
Add harness depends when present.

Include libyaml in all test builds.

Fix mode on paths before trying to remove and set test path with mode 770 to match the Perl test harness.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off), binReq (e.g. command/archive-get), or containerReq (e.g. protocol/protocol) keywords.

Builds and code generation need to be done in advance. The following commands are required for setup:

meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug build pgbackrest
ninja -C build test/src/test-pgbackrest
build/src/build-code help pgbackrest
build/src/build-code postgres pgbackrest

Now tests can be run, e.g.:

build/test/src/test-pgbackrest --module=postgres/interface
2022-07-20 19:02:14 -04:00