1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1821 Commits

Author SHA1 Message Date
David Steele
3a05359087 Create test matrix for mock/stanza to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
David Steele
6d3e18b181 Reduce expect log level in mock/stanza tests.
The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior.

Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit.

Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
David Steele
2f081f3ec7 Rename test modules for consistency.
The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
David Steele
d489eb87f7 Create test matrix for mock/archive to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests.  Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-23 15:59:39 +02:00
David Steele
4a7588e604 Create aliases for test VMs ordered by age.
This will allow for smarter allocation of tests in the next commit.
2019-02-23 15:13:23 +02:00
David Steele
59d7958914 Reduce expect log level in mock/archive tests.
The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior.

Now the archive code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit.

Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-23 15:05:06 +02:00
David Steele
a9cbf23f4c Improve error when hostname cannot be found in a certificate.
Update error message with the hostname and more detail about what went wrong.  Hopefully this will help in diagnosing certificate/hostname issues.

Suggested by James Badger.
2019-02-23 07:28:27 +02:00
David Steele
1f66bda02e Fix non-compliant JSON for options passed from C to Perl.
We have been using a hacked-up JSON generator to pass options from C to Perl since the C binary was introduced.  This generator was not very compliant which led to issues with \n, ", etc. inside strings.

We have a fully-compliant JSON generator now so use that instead.

Reported by Leo Khomenko.
2019-02-22 12:02:26 +02:00
David Steele
70c30dfb61 Disable test-level stack trace by default.
Detailed stack traces for low-level functions (e.g. strCat, bufMove) can be very useful for debugging but leaving them on for all tests has become quite burdensome in terms of time.  Complex operations like generating JSON on a large KevValue can lead to timeouts even with generous values.

Add a new param, --debug-trace, to enable test-level stack trace, but leave it off by default.
2019-02-22 11:40:30 +02:00
David Steele
ae86e6d5b2 Add missing ToLog() coverage to String, List, and PgControl.
Missing coverage is exposed in the next commit which disables test tracing by default.
2019-02-22 11:31:37 +02:00
David Steele
a7c8906581 Fix incorrect tags. 2019-02-21 18:49:02 +02:00
David Steele
e14c0eeb65 Use driver for remote protocol introduced in da628be8.
The remote protocol was calling into the Storage object but this required some translation which will get more awkward as time goes by.

Instead, call directly into the local driver so the communication is directly driver to driver.  This still requires resolving the path and may eventually have more duplication with the Storage object methods but it seems the right thing to do.
2019-02-21 16:20:46 +02:00
David Steele
b1eb8af7d5 Resolve storage path expressions before passing to remote.
Expressions such as <REPO:ARCHIVE> require a stanza name in order to be resolved correctly.  However, if the stanza name is passed to the remote then that remote will only work correctly for that one stanza.

Instead, resolved the expressions locally but still pass a relative path to the remote.  That way, a storage path that is only configured on the remote does not need to be known locally.
2019-02-21 15:40:21 +02:00
David Steele
b4d4680f8c Allow cfgExecParam() to exclude options.
It is useful to be able to exclude an option even if it is shared by both commands.
2019-02-21 14:40:35 +02:00
David Steele
be6a3f131e Improve null-handling of strToLog().
NULL was returning {"(null)"} which was comprehensible but not very pretty.  Instead return null on NULL.
2019-02-21 14:26:06 +02:00
David Steele
1fd89f05af Add kvKeyExists() to KeyValue object.
Check if a key exists even if the value is NULL, which is the same result for a missing key.
2019-02-21 14:16:17 +02:00
Stefan Fercot
80df1114bd Fix info command missing WAL min/max when stanza specified.
This issue was a result of STORAGE_REPO_PATH prepending an extra stanza when the stanza was specified on the command line.

The tests missed this because by some strange coincidence the WAL dirs were empty for each test that specified a stanza.  Add new tests to prevent a regression.

Fixed by Stefan Fercot.
2019-02-21 12:09:12 +02:00
David Steele
1519f5b045 Add storageHelperFree() to storage helper.
Free all cached objects in the storage helper, especially the stanza name.

This clears the storage environment for tests that switch stanza names or go from a stanza name to no stanza name or vice versa. This is only useful for testing right now, but may be used in the future for commands than act on multiple stanzas.
2019-02-21 11:40:30 +02:00
David Steele
b0b5989aca Migrate remote archive-get command to C.
All required protocol commands are implemented so this is mostly a matter of enabling the feature and updating expect logs.
2019-02-20 22:57:18 +02:00
David Steele
6866ff031a Add exists() to remote storage. 2019-02-20 22:43:02 +02:00
David Steele
71bc5697b1 Increase per-call stack trace size to 4096.
This was previously 256, which was too small to log protocol parameters.  Not only did this truncate important debug information but varying path lengths caused spurious differences in the expect logs.
2019-02-20 22:23:19 +02:00
David Steele
73be64ce49 Add separate archive-get-async command.
This command was previously forked off from the archive-get command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-02-20 15:52:07 +02:00
David Steele
e48b406cdc Add instructions for building the coverage report.
These are intended to be temporary until a fully automated report is developed.

Since we don't know when that will happen, at least make it easier to generate the current report.
2019-02-20 15:48:14 +02:00
David Steele
da628be8a8 Migrate remote command to C.
Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait.

Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object.

Update the remote storage objects to comply with the protocol changes and add the storage protocol handler.

Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
David Steele
d211c2b8b5 Fix possible truncated WAL segments when an error occurs mid-write.
The file write object destructors called close() and finalized the file even if it was not completely written.  This was an issue in both the C and Perl code.

Rewrite the destructors to simply free resources (like file handles) rather than calling the close() method.  This leaves the temp file in place for filesystems that use temp files.

Add unit tests to prevent regression.

Reported by blogh.
2019-02-15 11:52:39 +02:00
Stephen Frost
2cd204f380 Change execRead() to return a size_t.
execRead() should be returning a size_t, not a void. Thankfully, this isn't actually used and therefore shouldn't be an issue, but we should fix it anyway.

Contributed by Stephen Frost.
2019-02-12 14:59:51 +02:00
David Steele
a5f6f801d7 Begin v2.11 development. 2019-02-12 14:11:16 +02:00
David Steele
35903b94d9 v2.10: Bug Fixes
Bug Fixes:

* Add unimplemented S3 driver method required for archive-get. (Reported by mibiio.)
* Fix check for improperly configured pg-path. (Reported by James Chanco Jr.)
2019-02-09 19:52:31 +02:00
David Steele
51971617a8 Fix year on 2.09 release.
Reported by Achilleas Mantzios.
2019-02-09 19:01:16 +02:00
David Steele
057e2e2782 Add unimplemented S3 driver method required for archive-get.
This was not being caught because the integration tests for S3 were running remotely and going through the Perl code rather than the new C code.

Implement the exists method for the S3 driver and add tests to prevent a regression.

Reported by mibiio.
2019-02-09 18:57:30 +02:00
David Steele
6e88f93991 Fix check for improperly configured pg-path.
The check to verify that pg-path and data_directory are equal was not working because pg-path was getting overwritten with data_directory before validation took place.

Reported by James Chanco Jr.
2019-02-05 18:55:07 +02:00
David Steele
be8f31a952 JSON improvements.
Optimize the parser implementation and make the renderer more null tolerant.

Also make some string and variant constructors null tolerant.
2019-02-05 15:34:18 +02:00
David Steele
b3d1a2a802 FIx typos and missing debug param. 2019-02-02 15:18:10 +02:00
David Steele
b29a8dd9c5 Automatically adjust db-timeout when protocol-timeout is smaller.
This already worked in reverse, but this case is needed when a command that only uses protocol-timeout (e.g. info) calls a remote process where protocol-timeout and db-timeout can be set.  If protocol-timeout was set to less than the default db-timeout then an error resulted.
2019-02-02 15:03:19 +02:00
David Steele
abc613b454 Begin v2.10 development. 2019-02-02 14:50:24 +02:00
David Steele
a89a376119 v2.09: Minor Improvements and Bug Fixes
Bug Fixes:

* Fix issue with multiple async status files causing a hard error. (Reported by Vidhya Gurumoorthi, Joe Ayers, Douglas J Hunley.)

Improvements:

* The info command is implemented entirely in C.
* Simplify info command text message when no stanzas are present by replacing the repository path with "the repository".
* Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds. (Contributed by Douglas J Hunley.)
* Update address lookup in C TLS client to use modern methods. (Suggested by Bruno Friedmann.)
* Include Posix-compliant header for strcasecmp() and fd_set. (Suggested by ucando.)
2019-01-30 22:37:35 +02:00
David Steele
02dff53d45 Add openssl-devel install to documentation build in 711b3e67. 2019-01-30 22:02:42 +02:00
David Steele
fb342c3125 Add latex install to documentation build in 711b3e67. 2019-01-30 21:11:52 +02:00
David Steele
aa3e5b8c72 Allow primary gid for the test user to be different from uid.
Apparently up until now they have always been the same, which is pretty typical.  However, if they were not then ContainerTest.pm was not happy.
2019-01-30 17:03:17 +02:00
David Steele
711b3e67cb Add examples for building a documentation host. 2019-01-30 16:53:59 +02:00
David Steele
dada50ee26 Fix hard-coded repository path.
This prevented packages from being passed to the documentation unless they were in the /backrest directory on the host.

Also make the local path /pgbackrest instead of the deprecated /backrest.

Reported by Heath Lord.
2019-01-30 13:39:22 +02:00
David Steele
8634eb8c43 Include Posix-compliant header for fd_set.
Similar to f3ae3c4f, add Posix-compliant header to make FreeBSD happy.

Suggested by ucando.
2019-01-29 12:08:14 +02:00
David Steele
b5a103f2df Improve P/PP type macro handling.
Rather than create _P/_PP variants for every type that needs to pass/return pointers, create FUNCTION_*_P/PP() macros that will properly pass or return any single/double pointer types.

There remain a few unresolved edge cases such as CHARPY but this handles the majority of types well.
2019-01-28 22:33:29 +02:00
David Steele
8b2a344cfe Remove unused type parameter from FUNCTION_TEST_RETURN().
This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well.

The original intention was probably to allow logging of TEST return values but that never happened.
2019-01-28 15:06:28 +02:00
David Steele
7333b6302f Improve CONST type macro handling.
Rather than create a CONST_ variant for every type that needs to be returned const, create a FUNCTION_LOG_RETURN_CONST() macro that will return any type as const.
2019-01-28 09:44:10 +02:00
David Steele
d29aa61286 Allocate extra space for concatenations in the String object.
The string object was reallocating memory with every concatenation which is not very efficient.  This is especially true for JSON rendering which does a lot of concatenations.

Instead allocate a pool of extra memory on the first concatenation (50% of size) to be used for future concatenations and reallocate when needed.

Also add a 1GB size limit to ensure that there are no overflows.
2019-01-27 11:50:09 +02:00
David Steele
82c2d615b3 Move MACRO_TO_STR() to common/debug.h.
This macro is handy for constructing static message strings so move it where the core code can see it.
2019-01-27 11:34:12 +02:00
David Steele
4027123ef1 Update contributor name. 2019-01-27 08:01:17 +02:00
David Steele
8f6d324b2c Fix issue with multiple async status files causing a hard error.
Multiple status files were being created by asynchronous archiving if a high-level error occurred after one or more WAL segments had already been transferred successfully.  Error files were being written for every file in the queue regardless of whether it had already succeeded.  To fix this, add an option to skip writing error files when an ok file already exists.

There are other situations where both files might exist (various fsync and filesystem error scenarios) so it seems best to retry in the case that multiple status files are found rather than throwing a hard error (which then means that archiving is completely stuck).  In the case of multiple status files, a warning will be logged to alert the user that something unusual is happening and the command will be retried.

Reported by fpa-postgres, Joe Ayers, Douglas J Hunley.
2019-01-26 16:59:54 +02:00
David Steele
f3ae3c4f9d Include Posix-compliant header for strcasecmp().
gcc has apparently merged this function in string.h but Posix specifies that it should be in strings.h.  FreeBSD at at least is sticking to the standard.

In the long run it might be better to implement our own strcasecmp() function but for now just add the header.

Suggested by ucando.
2019-01-26 13:48:46 +02:00