1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1964 Commits

Author SHA1 Message Date
David Steele
d72d4415a7 Use contextParentIdx instead of a loop when moving a context.
contextParentIdx was introduced in 90709dfd to improve the performance of mem context frees. memContextMove() did not get the message, however, and continued to use a loop to find the mem context in the old parent.

Use contextParentIdx to find the mem context in the old parent to avoid a loop.
2021-08-23 08:27:07 -04:00
David Steele
3787cf7803 v2.35: Binary Protocol
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.

Bug Fixes:

* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)

Improvements:

* Binary protocol. (Reviewed by Cynthia Shang.)
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)

Documentation Bug Fixes:

* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)

Documentation Improvements:

* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)

Test Suite Improvements:

* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-08-23 06:52:51 -04:00
David Steele
ad5e063990 Update containers for new PostgreSQL releases.
Also add a package that is required for older Debian versions.
2021-08-19 17:25:57 -04:00
David Steele
590e759968 Rename stringz.h to stringZ.h.
This is more consistent with our file naming conventions.
2021-08-19 11:57:37 -04:00
David Steele
9b45df7057 Rearrange MemContext struct members to save space on 64-bit.
On 64-bit systems this saves a bit of space (10%) due to alignment. 32-bit systems see no benefit.

Also add tests for the individual struct sizes.
2021-08-19 11:12:56 -04:00
David Steele
d69a596358 Fix infinite loop in protocolServerProcess() on error.
The error was written to the client and then another command read. If the write did not fail then the loop would never exit.

Instead exit on any error that is not raised by the command handler as we can pretty safely assume this is an unrecoverable protocol error. The command handler might throw a protocol error itself, but this should be caught in the next read or write in the main loop.
2021-08-18 09:48:37 -04:00
David Steele
9a6afe3fc1 Fix EOF behavior of ioReadSmall() to match ioRead().
If the buffer was not full at EOF then ioReadSmall() would get stuck in an infinite loop. Instead, return on EOF even if the buffer is not full.

This is not an issue in released versions since ioReadSmall() is not being used.

Also fix a comment typo.
2021-08-18 09:32:20 -04:00
Cynthia Shang
eca2fc6958
Update config/parse test to use standard patterns. 2021-08-12 12:38:07 -04:00
Cynthia Shang
e17865a03a
Update protocol/protocol test to use standard patterns. 2021-08-12 11:57:17 -04:00
David Steele
a0bdfa436c
Log backup file total and restore size/file total.
The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.

Add size/file total log message to restore since it did not exist before.
2021-08-11 13:39:36 -04:00
David Steele
6789ec420e Add additional checks to performance/storage test.
The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".

To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
2021-08-10 10:37:37 -04:00
David Steele
c8492aac41 Cleanup inherited clients in protocol test harness.
Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.

We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.
2021-08-09 17:11:39 -04:00
Cynthia Shang
f653b59664
Update db/db test to use standard patterns. 2021-08-09 16:35:48 -04:00
David Steele
33775e53c9 Fix test added for lstComparatorZ() in 9abf6a27.
strcmp() returns < 0 and > 0 but these are not guaranteed to be -1 and 1.
2021-08-06 11:26:12 -04:00
David Steele
935de1e7a0
Only pass selected repo options to the remote.
Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.

Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
David Steele
74c0c44fc8 Migrate error code generation to C.
Parse src/build/error.yaml and write to src/config/error.auto.h and src/config/error.auto.c.
2021-08-02 18:32:11 -04:00
David Steele
930fee3a0c Move bldStrId() into a C file.
This function was included in a header but not declared inline, so linker errors happened when the header was included into more than one file.

Because of the setjmp() in TRY_BEGIN() it can't be inlined so put it in a C file.

Also add some missing headers.
2021-08-02 17:49:05 -04:00
David Steele
7542eadc9f Increase timeouts in storage/remote test.
There have been intermittent failures on f33 (with coverage) but not on u16 (without coverage).

Reproducing this reliably has been very difficult, so just try increasing the timeouts. This is based on the observation that tests with coverage take longer than tests without, which may lead the f33 tests to fail if CI is running slower than usual.

This will not increase the runtime of the test unless there is an error.
2021-08-02 14:48:31 -04:00
David Steele
4e71c077f1 Clean src path in test.pl before attempting vpath builds.
If configure/make has been run in the src path it can conflict with tests, which may require different build options.

Also add a comment when rebuilding for code generation.
2021-08-02 10:00:42 -04:00
David Steele
c192ec4561 Fix rsync include list removed from test.pl in 9ee9b1fa.
This caused the include list to be ignored and all files to be rsync'd, which worked but took a much longer.
2021-08-02 08:52:49 -04:00
David Steele
9abf6a2709 Add lstComparatorZ().
Works just like lstComparatorStr() but with zero-terminated strings.
2021-07-30 18:08:04 -04:00
David Steele
8bca6946b4 Add line and column to build yaml error messages.
This makes it much easier to debug errors in the yaml files.
2021-07-30 17:51:56 -04:00
David Steele
b47a07b8b9 Remove generated config.auto.c file.
This file duplicated the command list that already exists in parse.auto.c.

Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 14:40:30 -04:00
David Steele
e32f9e146b
Add check for test path inside repo path.
If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.

Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.
2021-07-29 11:09:51 -04:00
David Steele
f3f0c64a78 Complete migration of config code generation to C.
This was started in c5ae047e but did not include generation of parse.auto.c.

The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.

Option order resolution now includes cycle detection.
2021-07-28 19:59:23 -04:00
David Steele
52d4574460 Automatically generate option value StringIds in make build-config.
Remove strIdGenerate() since bldStrId() performs the same function without cluttering the core code. Since bldStrId() is intended to work in non-debug builds, move the validity checks for input strings out of the DEBUG block.

StringIds are generated as 5/6 bit, whichever is most efficient, for each option value. cfgOptionStrIdInternal() has been updated for this logic.
2021-07-26 12:02:31 -04:00
Cynthia Shang
eeaab6a3d1
Update config tests (except parse) to use standard patterns. 2021-07-23 16:18:50 -04:00
David Steele
947c7a84cb Refactor cmdLocal()/cmdRemote() to accept a ProtocolServer object.
This allows a local/remote to be started independently of server initialization, which will be useful for implementing new transport types, e.g. TLS.

Also remove some dead code in localTest.c.
2021-07-23 10:35:36 -04:00
David Steele
0999de0279 Move noop from protocolClientNew() to local/remote initialization.
protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()).

The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
2021-07-23 08:32:30 -04:00
Cynthia Shang
4c9ddf5ef2
Update storage tests to use standard patterns.
The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.

Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.

This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c
2021-07-22 18:17:08 -04:00
David Steele
3a37482de9 Update db test module error for newer libpq versions.
PostgreSQL 14 libpq throws a more detailed error, so adjust the regular expression to handle the old and the new errors.
2021-07-22 15:54:30 -04:00
David Steele
eb98b8d2db Fix typo. 2021-07-21 13:19:09 -04:00
David Steele
46992acd80 Update Vagrantfile box version. 2021-07-20 15:43:48 -04:00
Cynthia Shang
4992508764
Update command/restore test to use standard patterns. 2021-07-20 13:30:17 -04:00
David Steele
9ee9b1fad6 Remove test.pl --smart, --dev, and --dev-test options.
--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.

Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 12:01:10 -04:00
Cynthia Shang
1522cb4ed2 Allow NULL path in HRN_STORAGE_PATH_REMOVE() macro.
Update command/backup test to pass NULL where appropriate.
2021-07-19 15:21:40 -04:00
Cynthia Shang
a6cdf5b22f
Update command/repo test to use standard patterns. 2021-07-19 10:47:49 -04:00
Cynthia Shang
ebe5848494
Update command/local and command/remote tests to use standard patterns.
Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.
2021-07-19 09:51:11 -04:00
David Steele
c5ae047e76 Partial migration of config code generation to C.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.

This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.

The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
2021-07-18 19:02:01 -04:00
David Steele
6397d73535 Update test container OS versions.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.

Add --no-install-recommends to apt-get commands to save a bit of time and space.

Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.

Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 17:29:20 -04:00
David Steele
81602f1593 Remove test.pl --vm-host option.
This option was once used to optimize bin builds but has been defunct for some time.
2021-07-17 11:09:53 -04:00
Cynthia Shang
d57299ada8 Allow NULL path in TEST_STORAGE_LIST() macro. 2021-07-16 13:07:16 -04:00
Cynthia Shang
117ffe8896
Update command/help tests to use standard patterns.
In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.

Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.
2021-07-15 17:51:48 -04:00
Cynthia Shang
4ad0bbda53
Update command/backup tests to use standard patterns.
Includes backup and backupCommon tests.

Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.

String values were also removed where they were no longer needed.
2021-07-15 17:00:20 -04:00
David Steele
cdc89fc975 Allow NULL path in HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 16:05:32 -04:00
Cynthia Shang
c0feca4673 Add noParentCreate parameter to HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 13:31:54 -04:00
David Steele
4a6ca54b47 Remove last vestiges of key replacement missed in b270253a. 2021-07-15 13:19:49 -04:00
Stefan Fercot
563d117967
Allow restore --type=lsn.
The LSN recovery target was added in PostgreSQL 10 but did not get added to pgBackRest.
2021-07-15 13:02:41 -04:00
Stefan Fercot
8fac1ac50d
Loop while waiting for checkpoint LSN to reach replay LSN.
It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.

Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.
2021-07-15 07:45:06 -04:00
Cynthia Shang
a6691c6f61 Clarify restore --type behavior in command reference. 2021-07-14 16:06:42 -04:00