1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2279 Commits

Author SHA1 Message Date
k_zshiba
3ad588443b
Update test.pl --psql-bin option to match command-line help.
The option to specify the path to psql was shown in the command-line help as --psql-bin but the option was actually named --pgsql-bin.

Rename to match the help so they are consistent.
2022-11-14 12:47:27 +08:00
David Steele
58b3c91bab Add raw mode to CipherBlock to save space.
The magic in the header is only required so that command-line openssl will recognize the file as being encrypted. In cases where the encrypted data cannot be read with the command-line tool it makes sense to omit the header magic to save some space.

Unfortunately this cannot be enabled for file bundling because it would break backward compatibility. However, it should be possible to enable it for the combination of bundling and block incremental.
2022-11-10 10:28:49 +09:30
David Steele
c9db7bc274 Update cipherBlockNew() to allow optional parameters.
This simplifies calls a bit since digest is never passed and allows for new optional parameters.
2022-11-06 16:12:23 +09:30
David Steele
221db610d2 Shorten names in real/all integration test matrix.
This should allow one or two more parameters to be added without going to a new line, which keeps the matrix easier to read.
2022-10-18 18:02:17 +13:00
David Steele
fee38c2c7c Pass filters to remote storage as a handler array.
The prior code required coverage in the storage/remote module for all filters that could be used remotely.

Now the filter handlers are set at runtime so any filter list can be used with a remote. This is more flexible and makes coverage testing easier. It also resolves a test dependency.

Move the command/remote unit test near the end so it will have access to all filters without using depends.
2022-10-18 16:11:35 +13:00
David Steele
1730ef4ac3
Add noTruncate flag to storageNewWriteP().
This flag skips truncation when opening a file for write on drivers that support it, currently Posix and CIFS. This is convenient for cases where the file needs to be manipulated directly using the file descriptor. Using the file descriptor is not ideal and additional functionality should be added to the storage interface, but for now at least this avoids code duplication, especially on close which updates owners, the timestamp, syncs, etc.

The remote driver forbids no truncate because a file descriptor is never available for a remote storage write object.

Update two instances in the current code which benefit from this new functionality, but the primary reason for the change is to support more complex restore deltas in the upcoming block incremental feature.
2022-10-18 11:33:19 +13:00
David Steele
7967c750d8
Fix protocol error on short read of remote file.
If a remote file read was stopped before the read was complete or if an error occurred in the middle of the read then the protocol would end up in a bad state and produce this error:

ProtocolError: client state is 'data-get' but expected 'idle'

Prevent this by reading the rest of the file on close() or free() to leave the protocol in an idle state for the next command.

This was a possible issue for bundling because the amount to read is known in advance and therefore eof may not be reached. However, I was only able to reproduce this issue with unreleased code.

On error this issue would cause the original error to be lost. The process may still fail with this fix (if the error comes from another source) but hopefully we'll get better information about the original error.
2022-10-18 11:09:00 +13:00
David Steele
ddd966cadc
Use more generic descriptions for pg_start_backup()/pg_stop_backup().
The names were changed in PostgreSQL 15, so update the code and docs to make the naming more generic where needed to avoid using a version-specific name in the logs and documentation.
2022-10-18 09:59:15 +13:00
David Steele
65be4c64a9 Finalize catalog number for PostgreSQL 15 release. 2022-10-16 09:58:35 +13:00
David Steele
909be412f8 Swap command/backup and command/restore unit tests.
Logically restore belongs after backup and in a future commit restore will have a dependency on some backup objects.
2022-10-14 12:08:40 +13:00
David Steele
46a0af3531 Return binary as result from CryptoHash filter.
The prior result was hex-encoded, which is not optimal. This was legacy from the interface with Perl and then the JSON protocol. The new binary protocol natively supports binary so it makes sense to use it and convert to hex where needed.

A number of these hex conversions can now be removed but that will need to be handled in another commit.
2022-10-05 18:05:56 -10:00
David Steele
5602f179a1
Add varint-128 encode/decode to IoRead/IoWrite.
This makes it more efficient to read/write (especially read) varint-128 to/from IO.

Update the Pack type to take advantage of the more efficient read and remove some duplicate code.
2022-10-05 17:01:35 -10:00
David Steele
102ce5dee4
Add persistent reference list to manifest.
The reference list was previously built at load time from whichever references existed in the file list. This was sufficient since the list was for informational purposes only.

The block incremental feature will require a reference list that contains all prior backups, even those that are not explicitly referenced from the manifest. Therefore it makes sense to build and persist a manifest list rather than building it at load time.

This list can still be used for informational purposes, though it needs to be sorted since the list it sill built for older manifest versions and may not be in sorted order.

Add strLstFindIdx() to find references in the list.
2022-10-05 16:28:31 -10:00
David Steele
c647bcb509
Add manifest flags for file processing during backup.
The prior method was to check a combination of fields to determine if a file needed to be copied, delta'd, or resumed. This was complicated and ultimately imposed a limitation on the number of operations that could be performed.

Introduce copy, delta, and resume flags in the manifest to make it clearer which operations need to be performed and to reduce complex and duplicated logic.

This also allows zero-length bundled files to be completed during manifest build rather than later on during backup processing.
2022-10-05 13:14:15 -10:00
David Steele
1ea6a4142e
Improve manifest file updates.
The prior manifestFileUpdate() function was pretty difficult to use since all the parameters had to specified. Instead, pass a ManifestFile struct that has all members set as needed.

When new struct members are added the manifestFileUpdate() call sites will still need to be reviewed, but this should make the process of adding members a bit simpler.
2022-10-04 14:19:12 -10:00
David Steele
f981fb45d9
Do not store references for zero-length files when bundling.
This appears to have been an oversight in 34d6495. Storing the reference is not really correct since the file is not stored in a prior backup. It also uses more space.

There is no real harm in storing the reference, since it is always ignored on restore, but the code is simpler if the zero-length files can be dealt with during the manifest and don't need additional handling later on. This is also an important part of some upcoming optimizations.
2022-10-04 13:22:31 -10:00
David Steele
6e26860c26 Do not log bundle info when a file is delta matched from a prior backup.
It is possible to log the bundle info correctly but the information is useless with the backup reference, which does not appear until later. For now just omit the bundle info so we are not logging something incorrect.
2022-10-02 17:48:43 -10:00
David Steele
4722ad87a7 Add test for differential file bundles to the command/backup unit.
This test exposes a small logging issue. The bundle information for the matched delta on PG_VERSION is not correct. This issue will be fixed in the next commit.

The information stored in the manifest *is* correct so this bug is essentially cosmetic.
2022-10-02 17:41:31 -10:00
David Steele
ac99201c0e Add bufferSize to cvtUInt64FromVarInt128() to further limit reads.
The current call site, manifestFileUnpack(), does not know the total buffer size but the buffer has always been maintained in memory so there should be no corruption. However, there are upcoming use cases where the buffer will be read from IO, the buffer size will be known, and additional sanity checking on buffer overruns will be valuable.

Also rename params to align better with cvtUInt64ToVarInt128().
2022-10-02 17:32:48 -10:00
Reid Thompson
01b81f9d37
Move link creation to storage interface.
Direct link creation via Posix functions has been moved to the Posix driver.

This change allows adding SFTP softlink creation in the SFTP driver using the standard interface.
2022-10-01 15:26:44 -10:00
David Steele
2a4137ed2e Add zero-length chunked content test to common/ioHttp unit.
It seems wasteful to chunk content when there is nothing to send but make sure we handle it gracefully just in case a server decides to do it.
2022-09-25 16:15:23 -08:00
David Steele
64b64b614c Fix comment typo. 2022-09-25 12:30:30 -08:00
David Steele
f1e8e49fa9 Use large error/log buffers in test harness.
Ninja produces quite a bit of output so error messages are often truncated by the default error/log buffers. Use large buffers in the test harness to capture the error even when there is a lot of output.

Ninja has introduced a --quiet option, but it is currently too new to be in any of our test distributions.
2022-09-23 14:00:58 -07:00
David Steele
6b355806cc v2.41: Backup Annotations
Bug Fixes:

* Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
* Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)

Features:

* Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)

Improvements:

* Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
* Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
* Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
* Improve warning message on backup resume. (Suggested by Cynthia Shang.)

Documentation Improvements:

* Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
2022-09-19 10:08:10 -07:00
David Steele
c39c9f220e
Fix issue when listing directories recursively with a filter.
While recursing and filtering, if the last entry in a directory was another directory containing entries then the parent list would get freed too early, causing a double free error or segfault.

Fix by ensuring that the completed list is at the top of the stack before freeing it. This will defer freeing parent lists until the contents of paths have been processed.
2022-09-15 12:00:44 -07:00
David Steele
240cd755d1 Add mem context test missing from 0f7b6a33.
A coverage exception was added during development but was not removed before commit.

Remove the exception and add a test for coverage.
2022-09-14 10:06:06 -07:00
David Steele
8fb61a809d
Add FN_INLINE_ALWAYS macro.
Eliminate a lot of useless repetition for a commonly used pattern.
2022-09-08 18:36:03 -06:00
David Steele
3b5df1e089
Update archive.info timestamps after a successful backup.
Lifecycle policies can cause the archive.info file and its copy to be removed since they are only updated on a stanza-upgrade. Update the timestamps after a successful backup to prevent this.

This does not mean that lifecycle policies should be used as a replacement for expiration. However, in some cases there may be policies in place that are out of admin control. If the lifecycle expiration is less than pgbackrest expiration then corruption of the earliest backup will occur at the very least and there might be other corruption which would make the repo unrecoverable.
2022-09-02 10:03:18 -05:00
David Steele
0f7b6a3344
Skip mem context cleanup in CATCH_FATAL() block.
An error that gets raised all the way to the top TRY block might need to free a lot of resources and any of these callbacks could throw an error and mask the original error. In fact this is pretty likely since we are already in an error state. For example, the Db object will try to close the remote db connection, but if the protocol is in a bad state it will not be able to do so.

Solve this, for now, by not freeing memory or calling callbacks in the CATCH_FATAL() block. This gives us a better chance if being able to report the error without encountering another error first.

For the most part, we don't need to worry about freeing resources (file handles, TLS contexts, etc.) if the program is going to exit immediately. However, it is important to attempt to terminate all active protocol connections, which is done by protocolFree() in main() since the protocol objects live in the top context.

Another way to handle this would be to implement an error stack and that is probably something we will do in the future. But, in the case of a segfault the original error would still be lost. Yet another option would be to still do cleanup but defer it until after the CATCH_FATAL() block.
2022-08-31 12:44:55 -04:00
David Steele
eda7f81ee4
Fix incorrect time expiration being used for non-default repositories.
If a repo is not specified for the expire command then the lowest repo becomes the default. The repo-retention-full value for time was being retrieved from the default rather than a specific repo which led to an incorrect expiration being applied.

Get the value from the specific repo and add a test.

It would be better if the default repo could not be queried in this case but it is not clear how to do that since the repo option is valid for expire (unlike, e.g., archive-push).
2022-08-31 10:01:12 -04:00
David Steele
f1cb848591 Fix comment typos in command/expire unit test. 2022-08-30 18:04:32 -04:00
Stefan Fercot
db75ffd270
Support --set in JSON output for info command.
Allow detailed information (e.g. error list, tablespace list) in JSON output that is already available in text output with the --set option.
2022-08-25 10:12:22 -04:00
Stefan Fercot
381fd0a5a4
Backup key/value annotations.
Allow key/value annotations to be added with the backup command and added/modified/removed with the new annotate command.

Annotations can be viewed with the info command in text mode when --set is specified and are always included in JSON output.
2022-08-24 10:52:33 -04:00
David Steele
b05d31f531
Allow upload chunk size to be configured for object stores.
There are performance benefits to increasing the upload chunk size as long as the tradeoff with additional memory consumption is acceptable.

Make the chunk size configurable for S3, GCS, and Azure, but don't attempt to do any validation of the chunk size beyond some sane limits. The defaults remain as is for each storage type to avoid any unintentional regressions.
2022-08-24 09:33:26 -04:00
David Steele
ff1188f92d Allow quote tag in command-line help.
The quote tag will be used in an upcoming commit.
2022-08-22 13:51:05 -04:00
David Steele
82786da154
Do not allow CATCH() to catch a fatal error.
Catching individual fatal errors was only used in testing so the tests have been updated to use other errors instead. CATCH_FATAL() is now the only way to catch fatal errors.

This simplifies the logic a bit for upcoming changes to error handling and cleanup.

Also fix an issue where passing errorMessage() directly to THROW*() would attempt to copy the message buffer instead of preserving it, which is undefined behavior. Since there were no instances of this behavior before this commit, this was not a live bug.
2022-08-16 16:15:48 -04:00
David Steele
02665a5894 Update Minio test/documentation container version. 2022-08-02 11:05:31 -04:00
David Steele
eb287b18c8 Add profiling, performance, and optimization to C test harness.
All unit and performance tests are now built by the C harness.

Remove all unit/performance test build code from Perl.

Remove code from C harness that is no longer used. This code was included so the C harness could be run separately, but that is no longer needed with this full integration.
2022-07-29 10:31:36 -04:00
David Steele
1e83f2a022 Add coverage to C test harness.
Coverage reports are still generated in Perl, but all the settings have been added to the C harness to generate raw coverage data.
2022-07-28 14:53:48 -04:00
David Steele
c99ea54f17 Integrate C test harness with Perl test harness.
The C test harness is used for unit tests from the Perl harness where possible. Currently, unit tests can be run in the C harness when --no-coverage is specified and --profile is not specified.

C harness tests work on meson 0.45.

The C harness runs with valgrind by default. Valgrind can be disabled with --no-valgrind.

Also rebuild containers to add meson and update the documentation so that meson builds will work (even though we don't do them yet).
2022-07-27 10:32:32 -04:00
David Steele
79d9884141 Move release build check to src/meson.build.
This allows the C harness to perform release builds which are required for some tests.
2022-07-25 09:39:54 -04:00
David Steele
c267ba51b1
Move standby timeline check after checkpoint.
The standby timeline check was being performed using pg_control data loaded before the backup started. If the backup was started immediately after a promotion the standby might not have executed a checkpoint and written the new timeline to pg_control.

Instead perform the timeline check after the checkpoint is executed. This should ensure that the new timeline is in pg_control.
2022-07-22 16:24:55 -04:00
David Steele
cbbe93f592 Improve warning message on backup resume.
The prior warning made it sound as if some action was required on the part of the user.

The new message should make it clearer that this action will be performed by pgBackRest.
2022-07-22 14:45:38 -04:00
David Steele
4c47cc5d47 Remove Debian package patch now that it has been merged upstream. 2022-07-22 09:24:29 -04:00
David Steele
19d9941367 More C test harness improvements and CI.
Build pgbackrest binary and auto-generated code automatically.

Remove --module option and allow modules to run by parameter. This is less verbose and multiple modules can be run at a time.

Allow filtering of modules. Multiple tests can be passed as parameters and if the module ends in / it will be used as a prefix filter. For example, common/ will run all the common modules.

If a test errors the remaining tests will still run but the test process will eventually exit with an error.

CI tests are included but unit tests remain on the development branch.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off) or containerReq (e.g. protocol/protocol) keywords.

Building the C test harness has been simplified:

meson -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug test/build/none pgbackrest
ninja -C test/build/none test/src/test-pgbackrest

To run all modules:

test/build/none/test/src/test-pgbackrest test

Just the common/error module:

test/build/none/test/src/test-pgbackrest test common/error

All info modules:

test/build/none/test/src/test-pgbackrest test info/
2022-07-21 20:10:51 -04:00
David Steele
edfcf1652c Test Dockerfile improvements.
Add tzdata package so timezone tests in command/restore work correctly.

Mark default git path as safe. This is a security fix that is not applicable in this environment, but must be set.

Also remove package cleanup, which is inconvenient when new packages need to be installed. It makes sense for containers that will be downloaded from Dockerhub but not so much for a locally-maintained container.
2022-07-21 19:27:58 -04:00
David Steele
5e5b04be37 Fix common/lock test creating "750" path.
This was clearly an attempt to set the mode when creating a directory, but it never worked and instead created a "750" directory in the current working directory.

Detected when running in an environment where the current working directory was read-only.
2022-07-21 18:19:47 -04:00
David Steele
f9bbafbf3f C test harness improvements.
Add harness depends when present.

Include libyaml in all test builds.

Fix mode on paths before trying to remove and set test path with mode 770 to match the Perl test harness.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off), binReq (e.g. command/archive-get), or containerReq (e.g. protocol/protocol) keywords.

Builds and code generation need to be done in advance. The following commands are required for setup:

meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug build pgbackrest
ninja -C build test/src/test-pgbackrest
build/src/build-code help pgbackrest
build/src/build-code postgres pgbackrest

Now tests can be run, e.g.:

build/test/src/test-pgbackrest --module=postgres/interface
2022-07-20 19:02:14 -04:00
David Steele
c625f05a13 Unify code builder binaries into a single binary.
Creating new binaries was convenient at first but has now become a maintenance issue.

Solve this by combining that into a single binary that takes an additional parameter to indicate which code should be built.

Also clean up path handling to make it easier to build code from the command line.
2022-07-20 17:45:39 -04:00
David Steele
7eb5d67985 Add module harness and shim support to the C test harness.
This allows running unit tests up to common/ini, excluding common/debug-off and common/assert-off.
2022-07-19 18:03:39 -04:00