1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

762 Commits

Author SHA1 Message Date
David Steele
410a04a58e Allow arbitary InOut filters to be chained in IoFilterGroup.
If InOut filters were placed next to each other then the second filter would never get a NULL input signaling it to flush.  This arrangement only worked if the second filter had some other indication that it should flush, such as a decompression filter where the flush is indicated in the input stream.

This is not a live issue because currently no InOut filters are chained together.
2018-11-28 14:20:12 -05:00
David Steele
838cfa44b7 Allow arbitrary multiplier and flush character in IoTestFilterMultiply.
This allows for more complex test filter chains.

Rename from IoTestFilterDouble to reflect the new functionality.
2018-11-28 14:02:14 -05:00
David Steele
3e254f4cff Add IoFilter interface to CipherBlock object.
This allows CipherBlock to be used as a filter in an IoFilterGroup.  The C-style functions used by Perl are now deprecated and should not be used for any new code.

Also add functions to convert between cipher names and CipherType.
2018-11-28 12:42:36 -05:00
David Steele
c3a84ccae0 Merge cipher.h into crypto.h.
There is not enough in cipher.h to make it worth having a separate header.
2018-11-28 12:03:27 -05:00
David Steele
7df7cf6e6d Rename cipherBlock*() functions to cipherBlock*C().
Some of the old names conflict with the new functions that must be created to implement the filter. Rename these to cipherBlock*C() to indicate that they take C-style parameters.

These functions are only used by the Perl LibC code and will be removed or refactored eventually.
2018-11-27 22:49:24 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
7a1a688835 Add EOF detection to content read in HttpClient.
If the connection closed before all content was sent httpClientRead() would get stuck in an infinite loop waiting for it to arrive.

EOF should never be reached during content read so immediately error if EOF is detected.
2018-11-24 09:12:44 -05:00
Cynthia Shang
f4a1751abc Improve JSON to Variant conversion and add Variant to JSON conversion.
Add boolean and one-dimensional list types to jsonToKv().

Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.

Contributed by Cynthia Shang.
2018-11-23 16:02:33 -05:00
David Steele
beae375330 Enable S3 storage for archive-get command in C.
The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.

The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.
2018-11-23 12:18:07 -05:00
David Steele
b5690e21a4 Allow I/O read interface to explicitly request blocking reads.
TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.

Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.

In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.
2018-11-23 12:01:36 -05:00
David Steele
256b727a3d Add S3 storage driver.
Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-21 19:32:49 -05:00
David Steele
72252ed2a1 Add HttpClient object.
A robust HTTP client with pipelining support and automatic retries.

Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried.

Only the HTTPS protocol is currently supported.
2018-11-21 19:11:45 -05:00
David Steele
1dd06a6e46 Add TlsClient object.
A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.

This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.

Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-21 18:43:25 -05:00
David Steele
ac426bc456 New test containers with static test certificates.
Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.

The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.

Two new certificates have been defined. See test/certificate/README.md for details.

The old dynamic certificates will be retained until they are replaced.
2018-11-21 18:13:37 -05:00
David Steele
bc25db5667 Add interface objects for libxml2.
Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.

This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-20 20:40:11 -05:00
David Steele
f743d4e924 Add testRepoPath() to let C unit tests know where the code repository is located.
This allows a C unit test to access data in the code repository that might be useful for testing.

Add testRepoPathSet() to set the repository path.

In passing remove extra whitespace in the TEST_RESULT_VOID() macro.
2018-11-20 15:48:56 -05:00
David Steele
d0e553e4c2 Reduce debug level for infoIni() to test.
Getters should generally not be logging at debug or trace level since it clutters the log.

In passing move the destructor to the end of the file.
2018-11-16 16:05:06 -05:00
David Steele
04d9e4d5a8 v2.07: Automatic Backup Checksum Delta
Bug Fixes:

* Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)
* Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
* Fix error after log file open failure when processing should continue. (Reported by vthriller.)

Features:

* Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang.)

Improvements:

* Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)
2018-11-16 09:50:50 -05:00
David Steele
332a68ea8d Fix incorrect config constant introduced in 5e3b7cbe.
This commit introduced PGBACKREST_CONFIG_ORIG_PATH_FILE_STR as a String constant for PGBACKREST_CONFIG_ORIG_PATH_FILE but failed to get the value correct.

Also, no test was added for PGBACKREST_CONFIG_ORIG_PATH_FILE_STR to prevent regressions as there is for  PGBACKREST_CONFIG_ORIG_PATH_FILE.
2018-11-16 08:48:02 -05:00
David Steele
b6f7cbc315 Add destructors to IoRead and IoWrite objects.
These interfaces previously used the memory context of the object they were associated with and did not have their own destructors.

There are times when it is useful to free the interface without also freeing the underlying object so give IoRead and IoWrite their own memory contexts and destructors.

In passing fix a comment type in bufferRead.c.
2018-11-15 16:25:46 -05:00
David Steele
480e1da798 Add ioWriteFlush() to flush pending output.
By default the IoWrite object does not write until the output buffer is full but this is a problem for protocol messages that must be sent in order to get a response.

ioWriteFlush() is not called internally by IoWrite but can be used at any time to immediately write all bytes from the output buffer without closing the IoWrite object.
2018-11-14 08:53:42 -05:00
David Steele
14d2f8b86c Reword misleading message in stack trace when parameter buffer is full.
The prior message stated that there had been a buffer overrun which is not true since the code prevents that.

In fact, this message means the parameter buffer filled while building the parameter list. Rather than display a partial list we output this message instead.

Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:11:11 -05:00
David Steele
8c7e97a369 Clarify comment about main.c being excluded from unit testing.
Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:08:26 -05:00
David Steele
7310203491 Remove obsolete comments. 2018-11-14 07:42:28 -05:00
David Steele
a0b2af567a Add TEST_LOG() and TEST_LOG_FMT() macros.
These macros provide a convenient way to output debug information in tests.

They are not intended to be left in test code when it is committed to master.
2018-11-13 20:58:32 -05:00
David Steele
acb579c469 Tighten limits on code coverage context selection.
If the last } of a function was marked as uncovered then the context selection would overrun into the next function.

Start checking context on the current line to prevent this.  Make the same change for start context even though it doesn't seem to have an issue.
2018-11-13 10:37:58 -05:00
David Steele
086bc35ddc Make ioReadLine() read less aggressively.
ioReadLine() calls ioRead(), which aggressively tries to fill the output buffer, but this doesn't play well with blocking reads.

Give ioReadLine() an option that tells it to read only what is available.  That doesn't mean the function will never block but at least it won't do so by reading too far.
2018-11-12 21:18:53 -05:00
David Steele
a2672557f8 Allow storage path and file mode to be 0.
Not all storage drivers support Posix-style permissions, e.g. S3.
2018-11-12 20:00:57 -05:00
David Steele
7107cc68d2 Expand context shown in coverage and update colors.
Too few lines were shown for coverage context so show the entire function if it has any missing coverage.

Update colors to work with light and dark browser modes.
2018-11-12 18:11:16 -05:00
David Steele
aff9693e64 Add strEmpty() to String object.
strEmpty() is more concise and explanatory than strSize() == 0.
2018-11-12 14:13:55 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
8f857a975e Add constant macros to String object.
There are many places (and the number is growing) where a zero-terminated string constant must be transformed into a String object to be usable.  This pattern wastes time and memory, especially since the created string is generally used in a read-only fashion.

Define macros to create constant String objects that are initialized at compile time rather than at run time.
2018-11-10 09:37:12 -05:00
David Steele
df200bee2a Add regExpPrefix() to aid in static prefix searches.
The storageList() command accepts a regular expression as a filter.  This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code.  However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally.

S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible.

Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever.  It requires a ^ anchor and stops scanning when the first special character is found.
2018-11-09 16:50:22 -05:00
David Steele
8c504bd2f9 Improve alignment of expected vs. got error test results.
It is easier to compare the error messages when they start at the same column.
2018-11-09 10:37:59 -05:00
David Steele
b5a3c8c84b Add strCatChr() to String object. 2018-11-09 09:54:55 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
7cf1f42c5c Add time since the beginning of the run to each test statement.
Output the time in seconds of each test statement since the start of the test run.

This helps find individual tests that are running slowly.
2018-11-08 19:50:31 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
a9feaba9e5 Add memContextCallbackClear() to prevent double free() calls.
If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.

Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().
2018-11-07 08:51:32 -05:00
David Steele
48d2795f31 Merge crypto/random module into crypto/crypto.
There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.
2018-11-06 20:04:16 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
2cb312ef5a Add cryptoError() and update crypto code to use it.
This adds detail to error messages when available and improves code coverage.
2018-11-06 19:16:00 -05:00
David Steele
bef31f1802 Add base variants to all integer to string conversion functions.
Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.
2018-11-06 18:32:44 -05:00
David Steele
7de9584435 Ignore deleted files in rsync to test/repo.
Deleted files are showing up in git ls-files (added 57d78092) but they don't actually exist on disk.

If there is someway to exclude deleted files from ls-files then I can't find it, so just tell rsync to ignore missing files.
2018-11-04 08:39:48 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00