1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2531 Commits

Author SHA1 Message Date
David Steele
16ac5ee8d3 Rename block incremental manifest keys.
Since the keys need to be read/written in order, these keys make the logic a bit simpler.
2023-02-26 16:13:44 +07:00
David Steele
a9867cb0b8 Add repo-block-age-map and repo-block-size-map options.
Make these options configurable. This is primarily for testing purposes so the new options will be kept internal.
2023-02-26 14:49:34 +07:00
David Steele
dffc933384 Rename DeltaMap to BlockHash.
This more accurately describes what the object does.
2023-02-13 09:17:30 +07:00
David Steele
779efe0d7a Consistently declare block incremental size as size_t.
The block is often allocated in memory so size_t makes more sense than uint64_t.
2023-02-09 13:01:56 +07:00
David Steele
089fae035b Add block incremental to real/all test output. 2023-02-07 14:09:50 +07:00
David Steele
31cad5e09e Check for stray execute permissions in test.pl --code-format.
Sometimes these can get set while updating permissions from with a VM or container.

Also fix a few permissions that were not correct.
2023-02-01 12:57:04 +07:00
David Steele
c5907a2e71 Remove references to SSH made obsolete when TLS was introduced.
Also remove details about SSH compression that are not helpful.
2023-01-31 08:28:32 +07:00
David Steele
ce0ea2cfab Use uncrustify for code formatting.
The code is not completely reflowed yet so there are some cases that uncrustify will not catch. The formatting will be improved over time.

Some block of code require special formatting so have been surrounded with the {uncrustify-off}/{uncrustify-on} markers. These exceptions should be kept to a minimum.

Add --code-format (to reformat code) and --code-format-check (to check formatting) to test.pl.

Add a CI test that will check code formatting. Code must be correctly formatted before it can be merge to integration.

Add documentation to the coding standards for code formatting.
2023-01-30 12:22:41 +07:00
David Steele
d4070c9064 Reformat code with uncrustify.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:

* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.

Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.

Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
David Steele
b2202c36d9 Fix formatting errors.
Errors in our current (manually-maintained) code format discovered by uncrustify.
2023-01-30 11:16:31 +07:00
David Steele
f91af30584 Add FN_PRINTF() macro.
Bring the format(printf) attribute in line with the FN_NO_RETURN and FN_INLINE_ALWAYS macros.

This is simpler to read and can be customized for different compilers.
2023-01-30 10:51:48 +07:00
David Steele
f8ea51bf8f Refactor stackTraceToZ() to avoid multiple versions of the function.
stackTraceToZ() was split this way in c8264291 to allow complete coverage. 0becb6da added a shim to improve coveage but missed simplifying the function.
2023-01-30 10:27:17 +07:00
David Steele
053468bfb1 v2.44: Remove PostgreSQL 9.0/9.1/9.2 Support
Improvements:

* Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
* Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
* Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)

Documentation Improvements:

* Add warning about enabling "hierarchical namespace" on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
* Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
* Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
* Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
* Add dark mode to the website. (Suggested by Stephen Frost.)
2023-01-30 09:15:44 +07:00
David Steele
a28f3d49c2 Add --no-back-trace option to test.pl.
Running valgrind and backtrace together has been causing tests to timeout in CI, mostly likely due to limited resources. This has not been a problem in normal development environments.

Since it is still important to run backtraces for debugging, split the u22 test that was doing all this work to run coverage and backtrace together and valgrind-only as a separate test. As a bonus these tests run faster separately and since they run in parallel the total execution time is faster.
2023-01-28 18:55:53 +07:00
David Steele
16c625353d Skip test modules in coverage report.
Coverage of the test code is useful during development but it does not seem worth adding it to the core code coverage report.
2023-01-28 16:22:04 +07:00
David Steele
d5ef4ce747 Fix permissions on source files. 2023-01-28 11:01:29 +07:00
David Steele
1da2666a9e Add manifest test harness.
These macros make adding paths/files/etc to a manifest simpler and easier to read.
2023-01-21 14:03:27 +07:00
David Steele
912eec63bb
Block-level incremental backup.
The primary goal of the block incremental backup is to save space in the repository by only storing changed parts of a file rather than the entire file. This implementation is focused on restore performance more than saving space in the repository, though there may be substantial savings depending on the workload.

The repo-block option enables the feature (when repo-bundle is already enabled). The block size is determined based on the file size and age. Very old or very small files will not use block incremental.
2023-01-20 16:48:57 +07:00
David Steele
008a18555c Fix comment indentation and wrapping. 2023-01-20 15:44:01 +07:00
David Steele
bfc9a5c33e Fix incorrect FUNCTION_TEST_END() in hrnPgWalToBuffer(). 2023-01-14 18:01:48 +07:00
David Steele
34e4835ff3
Refactor common/ini module to remove callbacks and duplicated code.
The callbacks in iniLoad() made the downstream code more complicated than it needed to be so use an iterator model instead.

Combine the two functions that were used to load the ini data to remove code duplication. In theory it would be nice to use iniValueNext() in the config/parse module rather than loading a KeyValue store but this would mean a big change to the parser, which does not seem worthwhile at this time.
2023-01-12 21:24:28 +07:00
David Steele
9ca492cecf Audit mem contexts returned from functions into the calling context.
It is possible for functions to accidentally leak child contexts into the calling context, which may use a lot of memory depending on the use case and where it happens.

Use the function return type to determine what should be returned and error when something else is returned. Add FUNCTION_AUDIT_*() macros to handle exceptions.

This checking is only performed during unit tests on the code being covered by the specific unit test.

Note that this does not work yet for memory allocations, i.e. memNew(). These are pretty rare so are not as much of an issue and they can be added in the future.
2023-01-12 17:36:57 +07:00
David Steele
de1dfb66ca Refactor logging functions to never allocate memory.
Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case.

This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit.

Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
David Steele
0becb6da31 Enhance libbacktrace to handle incomplete stack traces.
This fills in backtrace info at the bottom of the call stack when the stack trace is incomplete due to testing. This does not affect release builds, which is why it did not make the first cut, but it turns out to be useful for testing and barely changes the release code (when we do release this).

The recursion test in common/error was simplified because it would now return a very large trace.
2023-01-12 10:22:26 +07:00
David Steele
9347313cbf Fix error detail output to stderr.
The error detail should be output when the error is an assert (this part was working) or the log level is at least debug. In cases where log-level-console was at least debug but log-level-stderr was not the detail was lost.

Improve the range checking to output error detail to stderr when log-level-console is at least debug.
2023-01-11 19:16:31 +07:00
David Steele
c826429123 Improve libbacktrace implementation.
The libbacktrace feature has not been working since the move to meson because libbacktrace detection was not added to the meson build. Add libbacktrace to meson and improve the feature so that it can be compiled into release builds.

The prior implementation fetched line numbers with each stack trace push. Not only was this slow but it missed any functions that were not being tracked on our stack.

Instead just examine the backtrace when an error happens and merge it with the info we have on our stack. If the backtrace is not available then the output remains as before.

Also remove --backtrace from test.pl since the library is now auto-detected.

Leave this library out of the production build for now to give it a little time to shake out in testing.
2023-01-11 11:19:26 +07:00
David Steele
6633ccd18d Do not output stderr on unit/integration test errors.
Since stderr is being redirected to stdout this results in duplication of the error output.
2023-01-05 14:03:43 +07:00
David Steele
4429bc82f5 Add unit tests for the unit test build code.
When this code was migrated to C the unit tests were not included because there were more important priorities at the time.

This also requires some adjustments to coverage because of the new code location.
2023-01-05 12:59:06 +07:00
David Steele
a5499afabc Add CHECK_FMT() macros and tests for CHECK*() macros.
The CHECK_FMT() macro will be used in some upcoming tests but could also be useful in the core code.

The CHECK*() macros never had tests so add them.
2023-01-05 12:44:00 +07:00
David Steele
d51a86c621 Add BUFFER_EXTERN() and cleanup hash zero constants.
BUFFER_EXTERN() provides a clean way to create buffer constants.

Convert HASH_TYPE_SHA256_ZERO_STR to HASH_TYPE_SHA256_ZERO_BUF to be consistent with HASH_TYPE_SHA1_ZERO_BUF.
2023-01-02 17:30:57 +07:00
David Steele
f018912908 Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN.
This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can).

Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
David Steele
4fb8a0ecdd Add meson unity build and tests.
This is immediately useful because it will detect any extern'd functions or variables that are not being used. It also detects functions or variables that are declared but not defined.

If a FV/VR_EXTERN macro is missing it will be detected either because of a mismatch in the declaration/definition or because a new defined symbol will appear in the nm test.

Eventually the unity build will be used to create a more optimized pgbackrest binary but that will need to wait.
2022-12-31 17:13:41 +07:00
David Steele
8aa2b101bb Put logging functions unused by release builds into DEBUG blocks.
Also inline some functions that are needed in non-DEBUG builds.
2022-12-31 16:41:01 +07:00
David Steele
2332ce8ffc Move storageHelperFree() to storageHelper test harness.
This function was only used for testing so move into a test harness.
2022-12-31 13:14:27 +07:00
David Steele
8b218158ae Move regExpMatchPtr()/regExpMatchStr() to build/common/regExp module.
Similar to b9be4fa5, these functions are not used by the core code so move them to the build module. The new implementation is a little less efficient but that is much less of a worry in the build/test code.

Also remove regExpMatchSize() since it was not longer needed.
2022-12-31 12:54:33 +07:00
David Steele
fa9d831f9f Move xmlNodeAttribute() to build/common/xml module.
Similar to b9be4fa5, this function was not used by the core code so move it to the build module.
2022-12-31 11:09:50 +07:00
David Steele
163a004f30 Move strReplace() and strUpper() to build/common/string module.
Neither of these functions were used by the core code. strReplace() is only used in the tests but it doesn't hurt to put it in build since the build code is not distributed.
2022-12-31 10:26:11 +07:00
David Steele
d517d4a328 Add explicit keyword for covered modules included in other modules.
This was done by checking the extension but it is possible to include a module that does not have a vendor or auto extension. Instead make it explicit that the module is included in another module.

Also change the variable from "include" to "included" to make it clearer what it indicates.
2022-12-31 10:10:44 +07:00
David Steele
cebbf0d012 Remove unused functions.
These functions were either added with the intention that they would be used or they became obsolete over time.
2022-12-30 16:26:48 +07:00
Stefan Fercot
b9be4fa540
Restore errors when no backup matches the current version of PostgreSQL.
It is probably not a good idea to restore the latest backup when it was not made from the current PostgreSQL version. If there is no backup after a stanza-upgrade then replicas might be built with a prior version leading to failures.

Add an error in this case if the latest backup would be used, i.e. --set or --type=time/lsn is not specified.
2022-12-29 15:37:27 +07:00
David Steele
44da314adb
Add compress-level range checking for each compress-type.
The prior range checking was done based on the valid values for gz. While this worked it was a subset of what is available for lz4 and zst.

Allow the range to be specified for each compress-type. Adding this functionality to the parse module would be a better solution but that is a bigger project than this fix deserves, at least for now.
2022-12-27 20:05:08 +07:00
David Steele
56b55f81e8
Add repository checksum to make verify and resume more efficient.
Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data.

This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum.

Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
David Steele
2ab845e263
Store manifest checksums in memory more efficiently.
Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions.

There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
David Steele
77c721eb63
Remove support for PostgreSQL 9.0/9.1/9.2.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.

Remove all logic associated with 9.0/9.1/9.2 and update the tests.

Document the new support policy.

Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
David Steele
010efffb0c Add hex encode/decoding to decode module.
This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
David Steele
9a9ee8e640 Rename EncodeType enum to EncodingType.
This avoids constructs such as decodeToBin(encodeBase64, ...) which are confusing since decode and encode are used in the same function call. decodeToBin(encodingBase64, ...) makes it clearer what is happening.
2022-12-11 18:55:25 +07:00
David Steele
1345caa3f7 Update missing flag in storage/posix test.
The storageNewItrP() permissions test was running twice with the errorOnMissing flag set to false. Fix by setting to true for one test.

Also update the comments to be clearer about what the tests are doing and fix minor formatting.
2022-12-07 20:30:38 +07:00
David Steele
4dc632d570 Add backup test harness.
This allows test backups to be run in other test modules.

It is likely that more logic will be moved here but for now this suffices to get test backups working in the restore module.
2022-12-05 14:15:15 +08:00
David Steele
c4bf775099
Fix missing reference in diff/incr backup.
When loading prior manifests without the new reference list, the code failed to add the current backup to the reference list. Since the current backup is never explicitly referenced, building references from the file list was not sufficient to generate a complete list.

The main problem here was a bad test, fixed in 28f6604. This masked the issue and prevented it from being found. Now it is clear in the test that the current label is missing from the reference list.

Fix by adding the current label to the reference list if a reference list is not stored in the manifest.
2022-11-28 16:42:35 +08:00
David Steele
28f660482a Move manifestBackupLabelSet() test in info/manifest unit.
Changing the label of a manifest that already had a label was not a good test and it ended up masking a bug where the current backup label was not being added to the reference list on manifest load, since manifestBackupLabelSet() added the label to the reference list. In fact, manifestBackupLabelSet() should never be called after a manifest load or even after the label has been set.

Add an assertion to prevent manifestBackupLabelSet() being called when the label is already set.

The bug exposed here will be fixed in a subsequent commit.
2022-11-26 10:48:02 +08:00
David Steele
3f363cb3ae
Add hint when an option is specified without an index.
Hopefully this will make it a little clearer to the user what is wrong when they specify an indexed option without an index.

Also fix an ambiguous use of cfgParseOptionP(). The prior code worked in that it set prefixMatch = true but it was not very readable.
2022-11-22 15:04:13 +08:00
David Steele
70b75532bf v2.42: Bug Fixes
Bug Fixes:

* Fix memory leak in file bundle backup/restore. (Reviewed by John Morris, Oscar. Reported by Oscar.)
* Fix protocol error on short read of remote file. (Reviewed by Stephen Frost.)

Improvements:

* Do not store references for zero-length files when bundling. (Reviewed by Stefan Fercot.)
* Use more generic descriptions for pg_start_backup()/pg_stop_backup(). (Reviewed by Greg Sabino Mullane, David Christensen. Suggested by Greg Sabino Mullane.)

Test Suite Improvements:

* Update test.pl --psql-bin option to match command-line help. (Contributed by Koshi Shibagaki. Reviewed by David Steele.)
2022-11-22 10:20:59 +08:00
k_zshiba
3ad588443b
Update test.pl --psql-bin option to match command-line help.
The option to specify the path to psql was shown in the command-line help as --psql-bin but the option was actually named --pgsql-bin.

Rename to match the help so they are consistent.
2022-11-14 12:47:27 +08:00
David Steele
58b3c91bab Add raw mode to CipherBlock to save space.
The magic in the header is only required so that command-line openssl will recognize the file as being encrypted. In cases where the encrypted data cannot be read with the command-line tool it makes sense to omit the header magic to save some space.

Unfortunately this cannot be enabled for file bundling because it would break backward compatibility. However, it should be possible to enable it for the combination of bundling and block incremental.
2022-11-10 10:28:49 +09:30
David Steele
c9db7bc274 Update cipherBlockNew() to allow optional parameters.
This simplifies calls a bit since digest is never passed and allows for new optional parameters.
2022-11-06 16:12:23 +09:30
David Steele
221db610d2 Shorten names in real/all integration test matrix.
This should allow one or two more parameters to be added without going to a new line, which keeps the matrix easier to read.
2022-10-18 18:02:17 +13:00
David Steele
fee38c2c7c Pass filters to remote storage as a handler array.
The prior code required coverage in the storage/remote module for all filters that could be used remotely.

Now the filter handlers are set at runtime so any filter list can be used with a remote. This is more flexible and makes coverage testing easier. It also resolves a test dependency.

Move the command/remote unit test near the end so it will have access to all filters without using depends.
2022-10-18 16:11:35 +13:00
David Steele
1730ef4ac3
Add noTruncate flag to storageNewWriteP().
This flag skips truncation when opening a file for write on drivers that support it, currently Posix and CIFS. This is convenient for cases where the file needs to be manipulated directly using the file descriptor. Using the file descriptor is not ideal and additional functionality should be added to the storage interface, but for now at least this avoids code duplication, especially on close which updates owners, the timestamp, syncs, etc.

The remote driver forbids no truncate because a file descriptor is never available for a remote storage write object.

Update two instances in the current code which benefit from this new functionality, but the primary reason for the change is to support more complex restore deltas in the upcoming block incremental feature.
2022-10-18 11:33:19 +13:00
David Steele
7967c750d8
Fix protocol error on short read of remote file.
If a remote file read was stopped before the read was complete or if an error occurred in the middle of the read then the protocol would end up in a bad state and produce this error:

ProtocolError: client state is 'data-get' but expected 'idle'

Prevent this by reading the rest of the file on close() or free() to leave the protocol in an idle state for the next command.

This was a possible issue for bundling because the amount to read is known in advance and therefore eof may not be reached. However, I was only able to reproduce this issue with unreleased code.

On error this issue would cause the original error to be lost. The process may still fail with this fix (if the error comes from another source) but hopefully we'll get better information about the original error.
2022-10-18 11:09:00 +13:00
David Steele
ddd966cadc
Use more generic descriptions for pg_start_backup()/pg_stop_backup().
The names were changed in PostgreSQL 15, so update the code and docs to make the naming more generic where needed to avoid using a version-specific name in the logs and documentation.
2022-10-18 09:59:15 +13:00
David Steele
65be4c64a9 Finalize catalog number for PostgreSQL 15 release. 2022-10-16 09:58:35 +13:00
David Steele
909be412f8 Swap command/backup and command/restore unit tests.
Logically restore belongs after backup and in a future commit restore will have a dependency on some backup objects.
2022-10-14 12:08:40 +13:00
David Steele
46a0af3531 Return binary as result from CryptoHash filter.
The prior result was hex-encoded, which is not optimal. This was legacy from the interface with Perl and then the JSON protocol. The new binary protocol natively supports binary so it makes sense to use it and convert to hex where needed.

A number of these hex conversions can now be removed but that will need to be handled in another commit.
2022-10-05 18:05:56 -10:00
David Steele
5602f179a1
Add varint-128 encode/decode to IoRead/IoWrite.
This makes it more efficient to read/write (especially read) varint-128 to/from IO.

Update the Pack type to take advantage of the more efficient read and remove some duplicate code.
2022-10-05 17:01:35 -10:00
David Steele
102ce5dee4
Add persistent reference list to manifest.
The reference list was previously built at load time from whichever references existed in the file list. This was sufficient since the list was for informational purposes only.

The block incremental feature will require a reference list that contains all prior backups, even those that are not explicitly referenced from the manifest. Therefore it makes sense to build and persist a manifest list rather than building it at load time.

This list can still be used for informational purposes, though it needs to be sorted since the list it sill built for older manifest versions and may not be in sorted order.

Add strLstFindIdx() to find references in the list.
2022-10-05 16:28:31 -10:00
David Steele
c647bcb509
Add manifest flags for file processing during backup.
The prior method was to check a combination of fields to determine if a file needed to be copied, delta'd, or resumed. This was complicated and ultimately imposed a limitation on the number of operations that could be performed.

Introduce copy, delta, and resume flags in the manifest to make it clearer which operations need to be performed and to reduce complex and duplicated logic.

This also allows zero-length bundled files to be completed during manifest build rather than later on during backup processing.
2022-10-05 13:14:15 -10:00
David Steele
1ea6a4142e
Improve manifest file updates.
The prior manifestFileUpdate() function was pretty difficult to use since all the parameters had to specified. Instead, pass a ManifestFile struct that has all members set as needed.

When new struct members are added the manifestFileUpdate() call sites will still need to be reviewed, but this should make the process of adding members a bit simpler.
2022-10-04 14:19:12 -10:00
David Steele
f981fb45d9
Do not store references for zero-length files when bundling.
This appears to have been an oversight in 34d6495. Storing the reference is not really correct since the file is not stored in a prior backup. It also uses more space.

There is no real harm in storing the reference, since it is always ignored on restore, but the code is simpler if the zero-length files can be dealt with during the manifest and don't need additional handling later on. This is also an important part of some upcoming optimizations.
2022-10-04 13:22:31 -10:00
David Steele
6e26860c26 Do not log bundle info when a file is delta matched from a prior backup.
It is possible to log the bundle info correctly but the information is useless with the backup reference, which does not appear until later. For now just omit the bundle info so we are not logging something incorrect.
2022-10-02 17:48:43 -10:00
David Steele
4722ad87a7 Add test for differential file bundles to the command/backup unit.
This test exposes a small logging issue. The bundle information for the matched delta on PG_VERSION is not correct. This issue will be fixed in the next commit.

The information stored in the manifest *is* correct so this bug is essentially cosmetic.
2022-10-02 17:41:31 -10:00
David Steele
ac99201c0e Add bufferSize to cvtUInt64FromVarInt128() to further limit reads.
The current call site, manifestFileUnpack(), does not know the total buffer size but the buffer has always been maintained in memory so there should be no corruption. However, there are upcoming use cases where the buffer will be read from IO, the buffer size will be known, and additional sanity checking on buffer overruns will be valuable.

Also rename params to align better with cvtUInt64ToVarInt128().
2022-10-02 17:32:48 -10:00
Reid Thompson
01b81f9d37
Move link creation to storage interface.
Direct link creation via Posix functions has been moved to the Posix driver.

This change allows adding SFTP softlink creation in the SFTP driver using the standard interface.
2022-10-01 15:26:44 -10:00
David Steele
2a4137ed2e Add zero-length chunked content test to common/ioHttp unit.
It seems wasteful to chunk content when there is nothing to send but make sure we handle it gracefully just in case a server decides to do it.
2022-09-25 16:15:23 -08:00
David Steele
64b64b614c Fix comment typo. 2022-09-25 12:30:30 -08:00
David Steele
f1e8e49fa9 Use large error/log buffers in test harness.
Ninja produces quite a bit of output so error messages are often truncated by the default error/log buffers. Use large buffers in the test harness to capture the error even when there is a lot of output.

Ninja has introduced a --quiet option, but it is currently too new to be in any of our test distributions.
2022-09-23 14:00:58 -07:00
David Steele
6b355806cc v2.41: Backup Annotations
Bug Fixes:

* Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
* Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)

Features:

* Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)

Improvements:

* Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
* Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
* Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
* Improve warning message on backup resume. (Suggested by Cynthia Shang.)

Documentation Improvements:

* Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
2022-09-19 10:08:10 -07:00
David Steele
c39c9f220e
Fix issue when listing directories recursively with a filter.
While recursing and filtering, if the last entry in a directory was another directory containing entries then the parent list would get freed too early, causing a double free error or segfault.

Fix by ensuring that the completed list is at the top of the stack before freeing it. This will defer freeing parent lists until the contents of paths have been processed.
2022-09-15 12:00:44 -07:00
David Steele
240cd755d1 Add mem context test missing from 0f7b6a33.
A coverage exception was added during development but was not removed before commit.

Remove the exception and add a test for coverage.
2022-09-14 10:06:06 -07:00
David Steele
8fb61a809d
Add FN_INLINE_ALWAYS macro.
Eliminate a lot of useless repetition for a commonly used pattern.
2022-09-08 18:36:03 -06:00
David Steele
3b5df1e089
Update archive.info timestamps after a successful backup.
Lifecycle policies can cause the archive.info file and its copy to be removed since they are only updated on a stanza-upgrade. Update the timestamps after a successful backup to prevent this.

This does not mean that lifecycle policies should be used as a replacement for expiration. However, in some cases there may be policies in place that are out of admin control. If the lifecycle expiration is less than pgbackrest expiration then corruption of the earliest backup will occur at the very least and there might be other corruption which would make the repo unrecoverable.
2022-09-02 10:03:18 -05:00
David Steele
0f7b6a3344
Skip mem context cleanup in CATCH_FATAL() block.
An error that gets raised all the way to the top TRY block might need to free a lot of resources and any of these callbacks could throw an error and mask the original error. In fact this is pretty likely since we are already in an error state. For example, the Db object will try to close the remote db connection, but if the protocol is in a bad state it will not be able to do so.

Solve this, for now, by not freeing memory or calling callbacks in the CATCH_FATAL() block. This gives us a better chance if being able to report the error without encountering another error first.

For the most part, we don't need to worry about freeing resources (file handles, TLS contexts, etc.) if the program is going to exit immediately. However, it is important to attempt to terminate all active protocol connections, which is done by protocolFree() in main() since the protocol objects live in the top context.

Another way to handle this would be to implement an error stack and that is probably something we will do in the future. But, in the case of a segfault the original error would still be lost. Yet another option would be to still do cleanup but defer it until after the CATCH_FATAL() block.
2022-08-31 12:44:55 -04:00
David Steele
eda7f81ee4
Fix incorrect time expiration being used for non-default repositories.
If a repo is not specified for the expire command then the lowest repo becomes the default. The repo-retention-full value for time was being retrieved from the default rather than a specific repo which led to an incorrect expiration being applied.

Get the value from the specific repo and add a test.

It would be better if the default repo could not be queried in this case but it is not clear how to do that since the repo option is valid for expire (unlike, e.g., archive-push).
2022-08-31 10:01:12 -04:00
David Steele
f1cb848591 Fix comment typos in command/expire unit test. 2022-08-30 18:04:32 -04:00
Stefan Fercot
db75ffd270
Support --set in JSON output for info command.
Allow detailed information (e.g. error list, tablespace list) in JSON output that is already available in text output with the --set option.
2022-08-25 10:12:22 -04:00
Stefan Fercot
381fd0a5a4
Backup key/value annotations.
Allow key/value annotations to be added with the backup command and added/modified/removed with the new annotate command.

Annotations can be viewed with the info command in text mode when --set is specified and are always included in JSON output.
2022-08-24 10:52:33 -04:00
David Steele
b05d31f531
Allow upload chunk size to be configured for object stores.
There are performance benefits to increasing the upload chunk size as long as the tradeoff with additional memory consumption is acceptable.

Make the chunk size configurable for S3, GCS, and Azure, but don't attempt to do any validation of the chunk size beyond some sane limits. The defaults remain as is for each storage type to avoid any unintentional regressions.
2022-08-24 09:33:26 -04:00
David Steele
ff1188f92d Allow quote tag in command-line help.
The quote tag will be used in an upcoming commit.
2022-08-22 13:51:05 -04:00
David Steele
82786da154
Do not allow CATCH() to catch a fatal error.
Catching individual fatal errors was only used in testing so the tests have been updated to use other errors instead. CATCH_FATAL() is now the only way to catch fatal errors.

This simplifies the logic a bit for upcoming changes to error handling and cleanup.

Also fix an issue where passing errorMessage() directly to THROW*() would attempt to copy the message buffer instead of preserving it, which is undefined behavior. Since there were no instances of this behavior before this commit, this was not a live bug.
2022-08-16 16:15:48 -04:00
David Steele
02665a5894 Update Minio test/documentation container version. 2022-08-02 11:05:31 -04:00
David Steele
eb287b18c8 Add profiling, performance, and optimization to C test harness.
All unit and performance tests are now built by the C harness.

Remove all unit/performance test build code from Perl.

Remove code from C harness that is no longer used. This code was included so the C harness could be run separately, but that is no longer needed with this full integration.
2022-07-29 10:31:36 -04:00
David Steele
1e83f2a022 Add coverage to C test harness.
Coverage reports are still generated in Perl, but all the settings have been added to the C harness to generate raw coverage data.
2022-07-28 14:53:48 -04:00
David Steele
c99ea54f17 Integrate C test harness with Perl test harness.
The C test harness is used for unit tests from the Perl harness where possible. Currently, unit tests can be run in the C harness when --no-coverage is specified and --profile is not specified.

C harness tests work on meson 0.45.

The C harness runs with valgrind by default. Valgrind can be disabled with --no-valgrind.

Also rebuild containers to add meson and update the documentation so that meson builds will work (even though we don't do them yet).
2022-07-27 10:32:32 -04:00
David Steele
79d9884141 Move release build check to src/meson.build.
This allows the C harness to perform release builds which are required for some tests.
2022-07-25 09:39:54 -04:00
David Steele
c267ba51b1
Move standby timeline check after checkpoint.
The standby timeline check was being performed using pg_control data loaded before the backup started. If the backup was started immediately after a promotion the standby might not have executed a checkpoint and written the new timeline to pg_control.

Instead perform the timeline check after the checkpoint is executed. This should ensure that the new timeline is in pg_control.
2022-07-22 16:24:55 -04:00
David Steele
cbbe93f592 Improve warning message on backup resume.
The prior warning made it sound as if some action was required on the part of the user.

The new message should make it clearer that this action will be performed by pgBackRest.
2022-07-22 14:45:38 -04:00
David Steele
4c47cc5d47 Remove Debian package patch now that it has been merged upstream. 2022-07-22 09:24:29 -04:00
David Steele
19d9941367 More C test harness improvements and CI.
Build pgbackrest binary and auto-generated code automatically.

Remove --module option and allow modules to run by parameter. This is less verbose and multiple modules can be run at a time.

Allow filtering of modules. Multiple tests can be passed as parameters and if the module ends in / it will be used as a prefix filter. For example, common/ will run all the common modules.

If a test errors the remaining tests will still run but the test process will eventually exit with an error.

CI tests are included but unit tests remain on the development branch.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off) or containerReq (e.g. protocol/protocol) keywords.

Building the C test harness has been simplified:

meson -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug test/build/none pgbackrest
ninja -C test/build/none test/src/test-pgbackrest

To run all modules:

test/build/none/test/src/test-pgbackrest test

Just the common/error module:

test/build/none/test/src/test-pgbackrest test common/error

All info modules:

test/build/none/test/src/test-pgbackrest test info/
2022-07-21 20:10:51 -04:00
David Steele
edfcf1652c Test Dockerfile improvements.
Add tzdata package so timezone tests in command/restore work correctly.

Mark default git path as safe. This is a security fix that is not applicable in this environment, but must be set.

Also remove package cleanup, which is inconvenient when new packages need to be installed. It makes sense for containers that will be downloaded from Dockerhub but not so much for a locally-maintained container.
2022-07-21 19:27:58 -04:00
David Steele
5e5b04be37 Fix common/lock test creating "750" path.
This was clearly an attempt to set the mode when creating a directory, but it never worked and instead created a "750" directory in the current working directory.

Detected when running in an environment where the current working directory was read-only.
2022-07-21 18:19:47 -04:00
David Steele
f9bbafbf3f C test harness improvements.
Add harness depends when present.

Include libyaml in all test builds.

Fix mode on paths before trying to remove and set test path with mode 770 to match the Perl test harness.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off), binReq (e.g. command/archive-get), or containerReq (e.g. protocol/protocol) keywords.

Builds and code generation need to be done in advance. The following commands are required for setup:

meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug build pgbackrest
ninja -C build test/src/test-pgbackrest
build/src/build-code help pgbackrest
build/src/build-code postgres pgbackrest

Now tests can be run, e.g.:

build/test/src/test-pgbackrest --module=postgres/interface
2022-07-20 19:02:14 -04:00
David Steele
c625f05a13 Unify code builder binaries into a single binary.
Creating new binaries was convenient at first but has now become a maintenance issue.

Solve this by combining that into a single binary that takes an additional parameter to indicate which code should be built.

Also clean up path handling to make it easier to build code from the command line.
2022-07-20 17:45:39 -04:00
David Steele
7eb5d67985 Add module harness and shim support to the C test harness.
This allows running unit tests up to common/ini, excluding common/debug-off and common/assert-off.
2022-07-19 18:03:39 -04:00
David Steele
8fdeed7807 Copy repository links as files for testing.
This makes the test code a bit simpler where we are listing a path but not following links.

Links in the repository can be used for testing but should never be committed to the main branch.
2022-07-19 17:57:13 -04:00
David Steele
eb77579008 Fix comment typo. 2022-07-19 17:25:07 -04:00
David Steele
1ff531090b v2.40: OpenSSL 3 Support
NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.

Improvements:

* OpenSSL 3 support. (Reviewed by Stephen Frost.)
* Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)
* Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)
* Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)
* Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris, Soulou. Suggested by Christian Montagne.)

Documentation Bug Fixes:

* Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete.)

Documentation Improvements:

* Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)

Test Suite Improvements:

* Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)
* Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
* Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
2022-07-18 09:32:30 -04:00
David Steele
364af1635d
Force target-timeline=current when restore type=immediate.
Explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository from this backup.

This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at least until they aren't really seen in the wild any longer.

PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly.
2022-07-14 08:26:03 -04:00
David Steele
75623d4583
Create snapshot when listing contents of a path.
Previously a callback was used to list path contents and if no sort was specified then a snapshot was not required. When deleting files from the path some filesystems could omit files that still existed, which meant the path could not be removed.

Filter . out of lists in the Posix driver since this special entry was only used by test code (and filtered everywhere in the core code).

Also remove callbacks from the storage interface and replace with an iterator that should be easier to use and guarantees efficient use of the snapshots.
2022-07-08 17:21:39 -04:00
David Steele
0eccbc8bf4 Meson builds work on version 0.45.
v0.45 ships with Ubuntu 18.04, which is currently the oldest distro we support. We may never do a Meson release on Ubuntu 18.04 but this allows us to start running unit tests with Meson in the meantime.

Some more granular options are not available so we use buildtype in more places.

The check for a in-tree autoconf/make build had to be removed since the filesystem APIs are not available.

Finally, alias_target was removed. This means that full paths must be used for build targets, which does not seem too bad. For instance, test/src/test-pgbackrest must now be used as a build target instead of simple test-pgbackrest.
2022-07-06 18:17:52 -04:00
David Steele
72960bbf17 Rename strTrunc() to strTruncIdx() and add strTrunc().
strTrunc() is now shorthand for truncating to index 0. This is convenient when a string is being reused.
2022-07-06 09:55:32 -04:00
David Steele
1c0bf0b15d Reorder null user/group checks in remote protocol and add tests.
Coverage for these checks was dependent on the order the files were read from disk, which made the tests fragile.

Rearrange the checks and add a test that won't depend on order.
2022-07-05 14:28:40 -04:00
David Steele
845d82f682
Use S3 IsTruncated flag to determine list continuation.
Previously we were just checking for the existence of NextContinuationToken, which the S3 documentation indicates will not be present when the list is not truncated. However, recent versions of Scality send a blank NextContinuationToken when IsTruncated is false. Sending the blank continuation token back causes Scality to send another blank continuation token and an infinite loop occurs.

Instead use IsTruncated (which is required to be present) to determine whether NextContinuationToken should be present. Error if NextContinuationToken is then missing or empty, since an empty token caused an infinite loop with the Scality server (which arguably should have errored when passed an empty token).
2022-07-01 20:32:10 -04:00
David Steele
61ca9b58a0 Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.
The TEST_STORAGE_LIST() macro is more robust and hides the callback mechanism from the caller.

Add features to TEST_STORAGE_LIST() that hrnStorageInfoListCallback() had.

Update tests to use the abbreviated type output (e.g. path/) generated by TEST_STORAGE_LIST().
2022-06-28 14:15:08 -04:00
David Steele
f863fc9888 Add experimental unit test harness written in C.
Having the test harness in C will allow us to remove duplicated Perl code and test on systems where Perl support is not present.

Custom harnesses and shims are currently not implemented, which means only the following tests in the common module will run: error, stack-trace, type-convert, assert-on, mem-context, time, encode, type-object, type-string, type-list, type-buffer, type-variant, reg-exp, log.

The experimental test harness is being committed with partial functionality so it can be used in Windows development. The remaining features will follow as needed.
2022-06-23 12:20:56 -04:00
David Steele
9fd85a104a Disable meson for all but debug builds.
The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.
2022-06-21 09:50:38 -04:00
David Steele
5ecae90f02 Use constants rather than replacements when possible in test.c.
Some of the replacements that were being done already existed as constants, so use the constants instead.

Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().
2022-06-17 16:31:48 -04:00
David Steele
dab1e4b6c6 Add strReplace().
Allows substrings to be replaced with another string.
2022-06-17 16:27:31 -04:00
David Steele
fb9acc1c93 Add higher level YAML functions.
These functions make parsing YAML simpler.
2022-06-17 16:17:02 -04:00
David Steele
55bcb933ee Move protocol module from common to command.
This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.

Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.
2022-06-17 11:17:52 -04:00
Stefan Fercot
eb72a80b47 Fix continuation character spacing. 2022-06-15 09:06:25 -04:00
David Steele
04f5ef25fa Add valgrind to Dockerfile missed in a16cf5ea. 2022-06-09 17:18:32 -04:00
David Steele
79443bea36 Move bldEnum() to the build/common/render module.
This function will be useful for other build/test modules so extern it.

Also skip the first upper-case when no prefix is provided.
2022-06-09 16:55:07 -04:00
David Steele
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
David Steele
08242ee6ac
OpenSSL 3 support.
There are two changes:

* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.

* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.
2022-06-06 14:47:47 -04:00
David Steele
f92ce674f7
Automatically create PostgreSQL version interfaces.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.

Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.

Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.

It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
2022-06-06 13:52:56 -04:00
David Steele
b8fc20d5b8
Add experimental Meson build.
Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.

The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.

Also patch the Debian package to force autoconf/make rather than Meson.
2022-06-03 14:13:56 -04:00
David Steele
148956aed8 Remove useless command/check unit test.
This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.
2022-06-01 10:13:57 -04:00
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
Reid Thompson
60d70fa66f
Disable incremental manifest save when resume=n.
The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.
2022-05-25 18:18:37 -04:00
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00
David Steele
4dcc9df222 Add labels in parse.auto.c to make diffs easier to read.
Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.

This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.

The change to parse.auto.c will be committed separately so it can be ignored in history/blame.
2022-05-16 09:14:46 -04:00
David Steele
901e829f6d v2.39: Verify and File Bundling
Bug Fixes:

* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)

Features:

* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)

Improvements:

* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)

Documentation Bug Fixes:

* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)

Documentation Improvements:

* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 08:46:24 -04:00
David Steele
19dd015d58
Fix issues in improved path validation for repo-* commands.
If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:

$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
ASSERT: [025]: start <= this->pub.size (on dev builds)
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)

Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.

Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.

This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.
2022-05-13 09:41:53 -04:00
David Steele
024500782e Reduce encrypted WAL segments sizes in command tests.
The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.

Reduce the size of the encrypted test segments where possible to improve overall test performance.
2022-05-12 17:02:08 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
Reid Thompson
a913113fda
Add option type descriptions.
This cuts down on repetition of the size descriptions and adds basic descriptions for the other option types.
2022-05-11 15:23:41 -04:00
David Steele
8ee85bc605 Fix "that that" typos. 2022-05-11 08:46:23 -04:00
David Steele
de816a0f57
Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
David Steele
3a40394472 Remove obsolete test in common/memContext.
Once upon a time the allocation array was allocated up front so this test was required for the top context, which did not allocate up front.

Now allocations are done on demand so this case is covered for every context that does not allocate memory.
2022-05-10 11:18:12 -04:00
Reid Thompson
6b98b3534e Fix typo. 2022-05-10 06:52:56 -04:00
David Steele
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
David Steele
39dddbb6bc Add limited CI for ppc64le/s390x using emulation.
The emulation is so slow that running all the unit tests would be too expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to test on one big-endian architecture to be sure that checksums are correct.

Update checksums in the tests where they had gotten out of date since the last time we were testing on s390x. Also use a different test in command/archivePushTest to show the name of the file when a checksum does not match to aid in debugging.

The command/archive-push test was updated but not included because there is also a permissions issue, which looks to be the same as what we see on MacOS/FreedBSD. Hopefully we'll be able to fix all of those at the same time.
2022-05-09 12:48:19 -04:00
David Steele
ef4c4ab852
Use variable instead of function to track FINALLY() state.
The function worked fine, but Coverity was unable to determine that the finally block was run, which led to false positives about unfreed memory.

Using a boolean in the block makes it clear to Coverity that the finally block will always be run no matter what else happens.

We'll depend on the compiler to optimize away the boolean if it is not used in a finally block. The cost of the boolean is fairly low in comparison to everything else being done in these macros, so it does not seem worth having a separate block even if the compiler is not able to eliminate the boolean.

This reverts most of 9a271e9 that fixed a bug caused by c5b5b58, which was also attempting to help Coverity understand FINALLY() blocks.
2022-05-09 10:39:43 -04:00
David Steele
46b7b72874 Add hint when unable to find the primary cluster.
If all available clusters are in recovery, pgBackRest will not be able to find a primary for the backup.
2022-05-06 18:23:36 -04:00
David Steele
53bfdbc01e Remove useless test in config/parse unit test.
Since the packSize field is 7 bits, it could never fail the check for > 127.

The compiler will catch any packs that are larger than 7 bits and then the pack size will need to be adjusted. For now just adjust the comment to reflect what the test does and give a clearer indication of what to do when a pack grows too large.
2022-05-06 16:02:44 -04:00
David Steele
77311a9af7 Fix indentation.
gcc11 complains about this indentation being misleading.
2022-05-06 15:38:03 -04:00
David Steele
efbcd975c4 Update Docker test image for Debian 9.
A change invalidated the current image which has been causing the d9 test to run longer.
2022-05-06 13:56:58 -04:00
David Steele
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
Reid Thompson
65d22e4325
Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
David Steele
e70c71049e Use uint8_t for optionResolveOrder.
This saves a bit of space and should not affect processing speed.

On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.
2022-05-06 07:49:23 -04:00
David Steele
808f7bf11c Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.
This test was likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:14:13 -04:00
David Steele
efe0a39a75 Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.
These tests were likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:01:02 -04:00
David Steele
876f3bbd1c Remove COLON_STR and separator parameter from cfgParseCommandRoleName().
The separator parameter in cfgParseCommandRoleName() was useless since it was always set to : and COLON_STR did not provide any clarity its the single other usage.
2022-05-05 18:15:05 -04:00
David Steele
5f8c9cd66a
Add ClockError for unexpected clock skew and timezone changes.
A distinct result code should help debugging of clock skew and timezone issues.
2022-05-05 10:19:11 -04:00
David Steele
b6bfd9f99d
Strip extensions from history manifest before showing in error message.
In cases where clock skew or timezone issues are preventing backup label generation the user could see an error like this:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I.manifest.gz'

This will happen if the most recent label is drawn from the history. It is cleaner (and probably less confusing) to strip off the extensions so the user sees:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I'
2022-05-05 09:20:49 -04:00
David Steele
ef672c74ad
Prevent memContextFree() from freeing memory needed by callbacks.
The order of callbacks and frees meant that memory needed during a callback (for logging in all known cases) might end up being freed before a callback needed it.

Requiring callbacks and logging to check the validity of their allocations is pretty risky and it is not clear that all possible cases have been accounted for.

Instead recursively execute all the callbacks first and then come back and recursively free the context. This is safer and it removes the need to check if a context is freeing so a simple active flag (in debug builds) will do. The caller no longer needs this information at all so remove memContextFreeing() and objMemContextFreeing().
2022-05-04 14:53:05 -04:00
Reid Thompson
d9088b2e2b
Show backup percent complete in info output.
In the JSON output the percent complete is storage as an integer of the percent complete * 100. So, before display it should be converted to double and divided by 100, or split using integer mod and div.

Note that percent complete will only be displayed on the host where the backup was executed. Remote hosts will show a backup/expire running with no percent complete.
2022-05-04 12:52:05 -04:00
David Steele
20782c88bc
PostgreSQL 15 support.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.

This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
2022-05-04 11:55:59 -04:00
David Steele
692fe496bd
Remove dependency on pg_database.datlastsysoid.
This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column.

Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available.

The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.
2022-05-04 08:22:45 -04:00
David Steele
9a271e925c
Fix error thrown from FINALLY() causing an infinite loop.
Any error thrown resets execution to the last setjmp(), which means that parts of the try block need to make sure they don't get run again. FINALLY() was not doing this so if it threw an error it would end up back in the FINALLY() block, where the error would likely be thrown again, causing an infinite loop.

Fix this by tracking the state of FINALLY() and only running it once. This requires cleaning the error stack like CATCH*() and clearing the error like TRY_END() depending on the order of execution.
2022-05-03 14:34:05 -04:00
David Steele
b89c568b5f Fix obsolete variable naming. 2022-05-03 10:50:48 -04:00
David Steele
9629908694
Error on all lock failures except another process holding the lock.
The archive-get/archive-push commands would not error for, .e.g permissions errors, when attempting to get a lock before launching the async process. Since the async process was not launched there would be no error status file and the user would get a generic failure message. Also, there would be no async log.

Refactor lockAcquireFile() to throw an error when failOnNoLock = false unless the file is locked by another process. This seems to be the original intent of this parameter and there may have been a mistake when porting from Perl. In any case it looks wrong enough to be considered a bug.
2022-05-03 10:13:32 -04:00
David Steele
eb435becb3 Exclude mem context name from production builds.
The mem context name is used to produce clearer debug errors but it has no purpose in production builds.

Also remove memContextName() and access the struct directly since the name is only used within the common/memContext module.

Note that a few errors that were thrown in production builds (and required the name) are now only thrown in debug builds. In practice we have not seen these errors in production builds due to extensive coverage so it does not seem worth modifying the error to work without the context name.

This saves some memory, which is worthwhile, but the goal is to refactor Strings and Variants to have their own mem contexts and this change will prevent them from using more memory than they are now, along with other changes that will be coming later.
2022-05-02 15:17:34 -04:00
David Steele
0055fa40fe Add user:group to lock permission error.
This will help debug permissions errors when the lock file cannot be created.
2022-05-02 09:45:57 -04:00
David Steele
03c71aa606 Add hint to check the log on archive-get/archive-push async error.
If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log.

This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
David Steele
4872a3f121 Improvements to test harness memory debugging.
Only set -DDEBUG_MEM for the modules currently being tested rather than globally.

Also run tests in a temp mem context. Running in the top context can confuse memory accounting when a new context is created in the top context.
2022-04-28 12:33:39 -04:00
David Steele
90f939b36f Fix leaks in common/io unit test.
These leaks make it harder to detect leaks in the core code, so fix them.
2022-04-28 12:31:59 -04:00
David Steele
8047e97e31 Fix leaked String and Variant in harnessPqScriptRun(). 2022-04-28 12:17:33 -04:00
David Steele
083c93eaa3 Reuse Strings in iniLoad().
Reuse the section/key/value Strings by truncating them instead of creating a new one every time.

Also add an error for empty sections. This function is only used for loading info files (not config files), which should never contain an empty section.
2022-04-28 10:11:15 -04:00
David Steele
bc46d4e37b Add cvtZSubNTo*() functions.
These functions allow conversion from substrings without needing to create a String or a temporary buffer.

httpDateToTime() no longer requires a temp mem context. Also improve handling of month search to avoid an allocation.

httpUriDecode() no longer requires a temp mem context.

jsonReadStr() no longer requires a temp mem context.

pgLsnFromWalSegment() no longer requires a temp mem context.

pgVersionFromStr() no longer requires a temp mem context. Also do a bit of refactoring.

storageGcsCvtTime() no longer leaks six Strings per call.

storageS3CvtTime() no longer leaks six Strings per call.
2022-04-28 09:50:23 -04:00
David Steele
3f7c8bc923 Fix object allocations in incorrect mem context in execOpen().
Object variables were begin allocated in the calling context rather than the object context.

This is not a live bug because Exec objects are currently created and opened in a long-lived context.
2022-04-26 10:15:47 -04:00
David Steele
41f9d69edc Combine functions in the command/stanza module into one function.
It is not clear why these were split out, but it probably had something to do with testing before storageList() could return NULL for an empty directory.

Also remove the tests that depended on a boolean return, which are no longer needed for coverage.
2022-04-25 15:38:49 -04:00
David Steele
582c3dab4c Add strLstAddSub*() and strLstAddSubZ*() functions.
These help with readability and remove a cause of leaks.
2022-04-25 12:32:33 -04:00
David Steele
ff45f463cf Use strLstAddZ() instead of strLstAdd() where possible.
Using STRDEF() to convert the zero-terminated string to a String has no performance advantage but generates more code.
2022-04-25 11:58:30 -04:00
David Steele
7900660d3a Add strLstNewFmt().
Simplifies adding a formatted string to a list and removes a common cause of leaks.
2022-04-25 11:47:43 -04:00
David Steele
45c3f4d53c
Improve JSON handling.
Previously read/writing JSON required parsing/render via a variant, which add many more memory allocations and loops.

Instead allow JSON to be read/written serially to improve performance and simplify the code. This also allows us to get rid of many String and Variant constant which are no longer required.

The goal is to be able to read/write very large (e.g. gigabyte manifest) JSON structures, which would not be practical with the current code.

Note that external JSON (GCS, S3, etc) is still handled using variants. Converting these will require more consideration about key ordering since it cannot be guaranteed as in our own formats.
2022-04-25 09:06:26 -04:00
David Steele
1e2b545ba4 Require type for FUNCTION_TEST_RETURN*() macros.
This allows code to run after the return type has been generated in the case where it is an expression.

No new functionality here yet, but this will be used by a future commit that audits memory usage.
2022-04-24 19:19:46 -04:00
David Steele
a2eee156b5 Fix instances where STRDEF() was used instead of STR().
In practice this didn't cause problems because the string buffer was still valid and strSize() was not being called.
2022-04-21 18:23:17 -04:00
David Steele
e18b70bf55 Allow *RETURN*() macros to accept struct initializers.
Struct initializers look like multiple parameters in a macro so use __VA_ARGS__ to reconstruct them.
2022-04-21 07:45:59 -04:00
David Steele
ea4d73f375 Fix ordering of backup-lsn-stop field in command/restore unit test.
All fields should be alphabetical. Currently the read code is tolerant of this, but that will not always be the case.

Fields are always written alphabetically so this is just a test issue introduced by d8d41321.
2022-04-20 19:56:26 -04:00
David Steele
cb7a5f1ef3 Add JSON error when value does not parse in Ini object.
If the JSON value fails to parse it is helpful to have the error message, at least for debugging.
2022-04-20 19:49:23 -04:00
David Steele
da6b4abc58 Handle missing archive start/stop in info/info backup unit test.
This is not a very realistic case since archive start/stop are always written, but it appears in many other unit tests so it should also be tested here.
2022-04-20 19:41:28 -04:00
David Steele
d897bf1ec2 Add size to info/manifest unit test.
This prevents the check from being order dependent.
2022-04-20 19:36:33 -04:00
David Steele
c304fafd45
Refactor PgClient to return results in Pack format.
Packs support stronger typing than JSON and are more efficient. For the small result sets that we deal with efficiency is probably not very important, but this removes another place where we are using JSON instead of Pack.

Push checking for result struct (e.g. single row) down into PgClient since it has easy access to this information rather than needing to parse the result set to find out.

Refactor all code downstream that depends on PgClient results.
2022-04-20 08:36:53 -04:00
David Steele
cfd6c7ceb4 Use specific integer types in postgres/client and db unit tests.
This will work better once we are able to transmit the results with stronger typing.

Also remove int2 which was not being used.
2022-04-18 12:14:22 -04:00
David Steele
9751ddc4f8 Update postgres/client unit test to conform to current patterns.
This includes adding test titles and using constants for query and error values that repeat.
2022-04-18 11:53:31 -04:00
David Steele
bc5f6fac34 Update postgres/client unit test for changes in libpq.
There have been some behavioral changes in libpq which require changes to the test.

Also update the instructions since it is now a bit easier to run against a real cluster.
2022-04-18 10:47:44 -04:00
David Steele
d103dd6238 Return stats as a JSON string rather than a KeyValue object.
There is no need to process the stats so a KeyValue is overkill.

Also remove the performance tests that check the stat totals since this is covered in the unit tests.
2022-04-14 20:34:42 -04:00
David Steele
e1ce731f8a Add test for protocol greeting when a field is missing.
A missing field and a NULL field are not exactly the same so it seems best to test both.

Because of the way KeyValue objects work the error is the same, but that will not always be true.
2022-04-14 19:37:03 -04:00
David Steele
aeecd07ad8 Fix reported error line number when ini key length is zero.
The line number was one less than it should have been, which could cause some confusion.

Since this only affected ini files with JSON values, which are always written programmatically, there is almost zero chance this has ever been a problem in the field.
2022-04-14 18:29:54 -04:00
David Steele
fa40bcdc5c
Throw error when unable to read lock process.
Previously the process id was skipped if it did not exist. Instead, throw an error and handle the errors in downstream code.

This was probably ignored at some point to provide backward-compatibility, but that is no longer required, if it ever was.
2022-04-11 14:08:16 -04:00
David Steele
79b2041663
Add lockRead*() functions for reading locks from another process.
Sometimes we need to read a lock from another process. This was done two different ways and in the case of cmdStop() was definitely hacky.

Centralize the logic to make it easier to read the locks for another process. This will also make it easier to add new lock data.
2022-04-08 15:55:41 -04:00
Reid Thompson
aad7171940
Suppress existing WAL warning when archive-mode-check is disabled.
When archive-mode-check is disabled and archive-push is running from multiple hosts, it is very likely that the file will already exist with the same checksum, so disable the warning.

However, if the checksums do not match, an error will still be thrown.
2022-04-08 15:00:20 -04:00
David Steele
4f543a4d67 Handle NULL path in TEST_STORAGE_LIST when remove is specified.
Using the path variable directly resulted in a path with (null) in it, which caused the remove to fail.

The pathFull variable already exists for this purpose so use it.
2022-04-08 11:07:26 -04:00
David Steele
571dceefec Add LENGTH_OF() macro.
Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times.

This macro makes the task easier, uses less space, and automatically adjusts when the type changes.
2022-04-07 19:00:15 -04:00
David Steele
8be11d32e4 Replace strCatFmt() with strCat()/strCatZ() where appropriate.
Most of these looked like copy/paste from a prior required strCatFmt() call.

There is no issue here since strCatFmt() works the same in these cases, but using strCat()/strCatZ() is more efficient.
2022-04-07 11:44:45 -04:00
David Steele
cff147a7d2
Add default for boolean options with unresolved dependencies.
If a boolean option had an unresolved dependency then the value would be NULL, which meant the dependency would need to be checked in the code to avoid an error. For example, cfgOptionBool(cfgOptOnline) needed to be checked before it was safe to call cfgOptionBool(cfgOptArchiveCheck).

Allow a default for boolean options when they are unresolved to simplify the code. This makes using the options easier and less prone to error. Not all boolean options get a dependency default in this commit, but more may be added in the future.
2022-04-06 14:45:51 -04:00
David Steele
5dba0d6e9b Set option-archive-copy flag in backup.manifest to false when offline.
In offline mode the pg_wal directory is copied, but that is not the same as archive-copy, which copies the exact set of WAL required from the archive.

This flag is purely for informational purposes so there is no live bug here, but the prior behavior was certainly misleading.
2022-04-05 18:42:19 -04:00
David Steele
54b4187527 Show Docker output when building containers if --log-level=detail.
This helps with debugging and monitoring container builds.
2022-04-05 13:14:42 -04:00
Reid Thompson
d8d4132118
Auto-select backup for restore command --type=lsn.
For PITR with --type=lsn, attempt to auto-select the appropriate backup set based on the --target LSN provided. Pick the most recent backup where backup-lsn-stop is less than or equal to the provided LSN.
2022-04-05 11:59:12 -04:00
David Steele
f60ec5055a Cleanup output to stderr in unit tests.
The unit tests were ignoring stderr but nothing being output there was important. Now a test will fail if there is anything on stderr.

This makes it easier to work with -fsanitize, which outputs to stderr.
2022-03-24 18:43:43 -06:00
David Steele
14016a86e7 Check that sha1 checksum is not empty in manifestFileUpdate().
The manifest test module was setting a blank value here and causing a stack overflow because memcpy() is used instead of strcpy().

This was really just a test issue but add an assert just in case the same were to happen in production code.

Also update a bogus checksum in the integration tests to the correct length to avoid running afoul of the assert.

Found with -fsanitize=address.
2022-03-24 13:13:35 -06:00
David Steele
75b26319ae Use strNewZ() in cases where STRDEF() assignment goes out of scope.
If a variable assigned with STRDEF() is referenced out of scope of the STRDEF() assignment then the value is undefined.

Luckily most of the instances are in tests but there is one in the core code. It is not clear if this is a live bug or not but it certainly needs to be fixed.

Found with -fsanitize=address.
2022-03-24 12:26:09 -06:00
David Steele
edf6c70baa Prevent signed integer overflow in cfgParseSize().
If the value and multiplier were large enough then the return value could overflow unpredictably.

Check the value to make sure it will not overflow with the current multiplier.

It would be better to present an "out of range" error to the user rather than "is not valid" but it doesn't seem worth the effort since the error is extremely unlikely.

Found with -fsanitize=undefined.
2022-03-24 11:00:51 -06:00
David Steele
ccbe2a1f70 Do not pass NULL to memcpy() in Buffer/String objects.
glibc and others seem tolerant of this but the behavior is undefined.

Found with -fsanitize=undefined.
2022-03-24 09:32:18 -06:00
David Steele
98792b1b0c Do not pass NULL to bsearch()/qsort() in List object.
glibc and others seem tolerant of this but the behavior is undefined.

Found with -fsanitize=undefined.
2022-03-24 09:22:05 -06:00
David Steele
424008d293 Allow files that become zero-length after the backup manifest is built.
It is possible that a file will be be truncated to zero-length after the backup manifest has been built. We could build logic into backupFile() to handle this case but it is hard to test well because of the race condition so tests would need to written directly against backupFile() and backupJobResult(). It hardly seems worth all that effort for a condition that occurs rarely, if ever.

Instead just remove the manifest check and add tests to restore to make sure it handles bundled zero-length files correctly. Logging will show that the file was bundled so if it happens a lot (which seems very unlikely) then we can think about an alternate implementation.
2022-03-23 10:41:36 -06:00
David Steele
7afaac0a3d
Allow repo-hardlink option to be changed after full backup.
This rule was added because there were not sufficient tests to demonstrate that the repo-hardlink option could be changed in a backup set.

Remove the restriction and add/update tests to show that it works.

This is necessary now because bundling requires that hardlinking be disabled. Rather than add code complexity, it seems better just to address this limitation.
2022-03-22 08:35:34 -06:00
Reid Thompson
5ae84d5e47
Improve path validation for repo-* commands.
Check for invalid path in repo-* commands. Perform path validation and throw an error when appropriate. Path may not contain '//'. Strip trailing '/' from path. Absolute path must fall under repo path.
2022-03-22 07:50:26 -06:00
nunopi
21cef09dfd
Add AWS IMDSv2 support.
IMDSv2 provides additional security to prevent instance metadata from being read by an attacker.

All AWS instances should provide IMDSv2 but still fail back to IMDSv1 if the IMDSv2 token request fails. This is in case there are any services outside AWS that are emulating IMDSv1 but have not implemented IMDSv2.
2022-03-16 11:02:29 -06:00
David Steele
2c96327e65 Remove extraneous double spaces in code and comments. 2022-03-15 17:55:48 -06:00
David Steele
3f66f42ef9
Rename bundle-* options to repo-bundle-*.
It seems best for these to be repo options so they can be configured per repo, rather than globally.

All clarify usage for repo-bundle-size and repo-bundle-limit.
2022-03-14 17:49:52 -06:00
Reid Thompson
7c9208ba85
Improve error message for invalid repo-azure-key.
Check that repo-azure-key is valid base64 when repo-azure-key-type = shared.
2022-03-11 10:10:02 -06:00
David Steele
0054677147 Add bundle logging to backup command.
This was added to the restore command so add it to the backup command as well.
2022-03-09 15:34:15 -06:00
David Steele
dca6da86bf
Optimize restore command for file bundling.
Since files are stored sequentially in a bundle, it is often possible to restore multiple files with a single read. Previously, each restored file required a separate read. Reducing the number of reads is particularly beneficial for object stores, but performance should benefit on any file system.

Currently if there is a gap then a new read is required. In the future we might set a limit for how large a gap we'll skip without starting a new read.
2022-03-09 15:03:28 -06:00
Reid Thompson
f7ab002aa7
Improve stop command to honor stanza option.
Improve the stop command, when force and stanza options are specified, to terminate only processes holding lock files for the given stanza. Prior to these changes, termination of all processes holding lock files regardless of stanza occurred.
2022-03-08 12:18:23 -06:00
David Steele
514137040e Add limit parameter to ioCopyP().
Allows the number of bytes copied to be limited.
2022-03-08 08:23:31 -06:00
Reid Thompson
330e19900e
Increase precision of percent complete logging for backup and restore.
For very large backups only getting an update per percent may not be often enough.

Add hundredths to the percent complete logging to provide more timely information.
2022-03-06 13:01:24 -06:00
David Steele
8f23b46b4b Replace percentage and size with a constant in restore test logs.
Checking percentage and size in every test can cause quite a bit of churn when changes are made.

Follow the example of the backup tests and replace percentage and size after the few tests to reduce churn.
2022-03-06 11:57:20 -06:00
David Steele
4d2fef1c37 Remove redundant restoreFile() test and improve coverage.
These tests were written before the restore command was fully migrated to C so many of them have become redundant.

In the cases were they still provide coverage, add tests to synthetic restores to replace them. In general, these higher level tests provide better coverage than poking at the restoreFile() function directly.
2022-03-06 11:48:22 -06:00
David Steele
5249b89a2e v2.38: Minor Bug Fixes and Improvements
IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported.

Bug Fixes:

* Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.)
* Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.)

Features:

* Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.)
* Add archive-missing-retry option. (Reviewed by Stefan Fercot.)
* Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)

Improvements:

* Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.)
* Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.)
* Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.)
* Do not retry fatal errors. (Reviewed by Reid Thompson.)
* Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.)
* Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.)

Documentation Bug Fixes:

* Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.)
* Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.)
* Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.)

Documentation Improvements:

* Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.)
* Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
2022-03-06 10:30:59 -06:00
David Steele
59a5373cf8
Handle TLS servers that do not close connections gracefully.
Some TLS server implementations will simply close the socket rather than correctly closing the TLS connection. This causes problems when connection: close is specified with no content-length or chunked encoding and we are forced to read to EOF. It is hard to know if this is a real EOF or a network error.

In cases where we can parse the content and (hopefully) ensure it is correct, allow the closed socket to serve as EOF. This is not ideal, but the change in 8e1807c means that currently working servers with this issue will stop working after 2.35 is installed, which seems too risky.
2022-03-02 11:38:52 -06:00
David Steele
fb5051fde7 Use vagrant user in the Docker container.
This is a bit of legacy from the current Vagrant environment used to do the release, but since it is not as easy to change the user in Vagrant, just make the Docker environment conform.

This allows documentation to be built in a Vagrant environment (or any environment with the same user name) and to be deployed in a Docker environment.
2022-02-26 13:50:30 -06:00
David Steele
b33cabe08c
Allow case-insensitive matching of HTTP connection header values.
The specification allows values for the connection header to be case-insensitive. See https://www.rfc-editor.org/rfc/rfc7230#section-6.1.
2022-02-25 10:51:40 -06:00
David Christensen
6320712323
Automatically strip trailing slashes for repo-ls paths.
Trailing slashes in at least some of the repository storage types were preventing repo-ls from displaying any content (presumably due to storage-specific behavior).

Since the path with the slash should be equivalent to the path without the slash, just remove it if provided by the user.
2022-02-23 13:53:02 -06:00
David Steele
53f1b25204
Improve validation of zero pages.
Checking that pd_upper == 0 is not enough since this field may be corrupted. Still use pd_upper as a quick check, but when it is zero proceed to check the rest of the page to ensure it is also all zeroes.
2022-02-23 13:17:14 -06:00
David Steele
9eec98c613
Retry on page checksum validation failure during backup.
Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL.

Also remove tests made redundant by the test merge in b4897077.
2022-02-23 12:05:53 -06:00
David Steele
67bdf07e69 Add XML to invalid XML error message.
There have been cases where pgBackRest has failed on invalid XML but it is not possible to determine what was wrong with the XML.

This will only work for XML up to about 8KiB (which is the error message limit) but it should work in most cases.
2022-02-23 10:26:39 -06:00
David Steele
10038db9c9
Add archive-missing-retry option.
Retry a WAL segment that was previously reported as missing by the archive-get command. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.

Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.

When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty.
2022-02-23 09:14:27 -06:00
David Steele
e6e1122dbc Pass file by reference in manifestFileAdd().
Coverity complained that this pass by value was inefficient:

CID 376402:  Performance inefficiencies  (PASS_BY_VALUE)
Passing parameter file of type "ManifestFile" (size 136 bytes) by value.

This was completely intentional since it gives us a copy of the struct that we can change without bothering the caller. However, updating fields is fine and may benefit the caller at some future data, and in any case does no harm now.

And as usual it is easier not to fight with Coverity.
2022-02-20 16:45:07 -06:00
David Steele
b489707793 Move command/backup-common tests in the command/backup module.
As much as possible it is better to get coverage with more realistic tests. Merging these modules will allow the page checksum code to be covered with real backups.
2022-02-18 17:50:05 -06:00
David Steele
efc09db7b9
Limit files that can be bundled.
Limit which files can be added to bundles, which allows resume to work reasonably well. On resume, the bundles are removed and any remaining file is eligible to be to be resumed.

Also reduce the bundle-size default to 20MiB. This is pretty arbitrary, but a smaller default seems better.
2022-02-17 07:25:12 -06:00
David Steele
34d649579e
Bundle files in the repository during backup.
Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest.

Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented.

Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups.

Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example.

Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling.

One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
David Steele
8046f06307
Do not retry fatal errors.
There is some evidence that retrying fatal errors, especially out of memory errors, may cause lockups. It makes sense to report fatal errors as quickly as possible and bypass retries. This may or not fix the lockup issue but it is worth doing either way.

For now, the only fatal errors will be AssertError and MemoryError.
2022-02-14 11:07:02 -06:00
David Steele
8d0cce66f8 Use normal error for protocol module error retry test.
Asserts will not be retried in a future commit, so adjust this test now to use non-assert errors.
2022-02-13 15:19:31 -06:00
David Steele
8573a2df14 Improve protocol module error test for protocolClientFree().
Using an assert here was never ideal and won't work once we start handling fatal errors differently.
2022-02-13 15:11:59 -06:00
David Steele
551e5bc6f6
Retry errors in S3 batch file delete.
If the entire batch failed it would be retried, but individual file errors were not retried. This could cause pgBackRest to terminate during expiration or when removing an unresumable backup.

Rather than retry the entire batch, delete the errored files individually to take advantage of the HTTP retry rather than adding a new retry loop. These errors seem rare enough that it should not be a performance issue.
2022-02-11 08:11:39 -06:00
Stefan Fercot
b26097f8d8
Add backup type filter to info command.
Support --type option in the info command to display only a specific backup type.
2022-02-09 10:18:39 -06:00
David Steele
cb630ffe3b
Remove logic that tried to determine additional file system compression.
In theory, the additional stat() call after a file has been copied to the repo can determine if additional compression has been applied by the file system. However, it has been a very long time since we tested this in practice. There are currently no unit tests that accurately test this feature since it requires a compressed file system like ZFS to work, which never seemed worth the extra cost.

It can also add a lot of time to backups if there are a large quantity of small files.

In addition, it stands as a blocker for combining files for small file support since it is no longer possible to get per-file sizes from the viewpoint of the file system. There are several ways this could be reworked but none of them are easy while at the same time maintaining current info functionality.

It doesn't seem worth keeping an untested feature that will only work in some special cases (if it still works) when it is blocking development.
2022-02-09 09:32:23 -06:00
David Steele
b1da4e84e8 Revert Minio to prior release.
The most recent release of Minio has broken CI builds but there is no logging to indicate what is wrong.

For now, just use the prior release to get CI builds working again. This kind if breakage is not uncommon for Minio but they usually resolve it in the next release.
2022-02-02 14:39:39 -06:00
David Steele
9b2f10dbb4
Refactor lock code.
Update lock code to use standard common/io functions and module patterns. This module was developed before the common/io module existed and our patterns had stabilized.
2022-01-31 16:48:28 -06:00
David Steele
22734eb376 Add ioBufferReadNewOpen() and ioBufferWriteNewOpen().
These are convenience functions to make the code a bit more compact where possible.
2022-01-31 10:03:56 -06:00
David Steele
cf5b3a302f Fix language in rh7 test container for aarch64.
The /etc/profile.d/lang.sh script was causing issues but it does not exist on amd64, so it seems the easiest thing was to remove it.

Fix how 32-bit VMs are determined now that another 64-bit architecture has been added.

And remove some obsolete VM hashes.
2022-01-26 13:22:31 -06:00
David Steele
e4df5b7d38
Simplify manifest file defaults.
Previously manifest load required two passes through the file list, one to load the data and one to set the defaults. This required each file to be packed twice.

Instead simply note that the file value is default and then set the file defaults when they are loaded from the manifest. This is made possible by the different internal/external representations for files so the same method cannot be applied to paths and links.

This change seems to resolve the performance issues noted in 61ce586 but there is no obvious reason why.
2022-01-24 15:21:07 -06:00