1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

988 Commits

Author SHA1 Message Date
David Steele
c3a84ccae0 Merge cipher.h into crypto.h.
There is not enough in cipher.h to make it worth having a separate header.
2018-11-28 12:03:27 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
Cynthia Shang
f4a1751abc Improve JSON to Variant conversion and add Variant to JSON conversion.
Add boolean and one-dimensional list types to jsonToKv().

Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.

Contributed by Cynthia Shang.
2018-11-23 16:02:33 -05:00
David Steele
e641c130d3 Fix release note typo. 2018-11-23 12:19:56 -05:00
David Steele
beae375330 Enable S3 storage for archive-get command in C.
The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.

The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.
2018-11-23 12:18:07 -05:00
David Steele
b5690e21a4 Allow I/O read interface to explicitly request blocking reads.
TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.

Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.

In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.
2018-11-23 12:01:36 -05:00
David Steele
256b727a3d Add S3 storage driver.
Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-21 19:32:49 -05:00
David Steele
72252ed2a1 Add HttpClient object.
A robust HTTP client with pipelining support and automatic retries.

Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried.

Only the HTTPS protocol is currently supported.
2018-11-21 19:11:45 -05:00
David Steele
1dd06a6e46 Add TlsClient object.
A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.

This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.

Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-21 18:43:25 -05:00
David Steele
ac426bc456 New test containers with static test certificates.
Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.

The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.

Two new certificates have been defined. See test/certificate/README.md for details.

The old dynamic certificates will be retained until they are replaced.
2018-11-21 18:13:37 -05:00
David Steele
53e3651cca Remove embedded semicolon from String constant macros.
The embedded semicolon led to inconsistent semicolons when using the macro and is not our general convention.

Remove embedded semicolons from the macros and add semicolons in usage where they were not present.
2018-11-21 08:30:14 -05:00
David Steele
bc25db5667 Add interface objects for libxml2.
Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.

This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-20 20:40:11 -05:00
David Steele
6680130c6f Require S3 key options except for local/remote commands.
S3 key options (repo1-s3-key/repo1-s3-key-secret) were not required which meant that users got an ugly assertion when they were missing rather than a tidy configuration error.

Only the local/remote commands need them to be optional.  This is because local/remote commands get all their options from the command line but secrets cannot be passed on the command line.  Instead, secrets are passed to the local/remote commands via the protocol for any operation that needs them.

The configuration system allows required to be set per command so use that to improve the error messages while not breaking the local/remote commands.
2018-11-20 19:24:53 -05:00
David Steele
f743d4e924 Add testRepoPath() to let C unit tests know where the code repository is located.
This allows a C unit test to access data in the code repository that might be useful for testing.

Add testRepoPathSet() to set the repository path.

In passing remove extra whitespace in the TEST_RESULT_VOID() macro.
2018-11-20 15:48:56 -05:00
David Steele
b0659278cc Add ServiceError for errors from a service that can be retried.
An example is HTTP 5xx errors which should mostly be retried.
2018-11-16 17:22:22 -05:00
David Steele
3aa4fda573 Add missing LOG_DEBUG() macro. 2018-11-16 16:17:24 -05:00
David Steele
d0e553e4c2 Reduce debug level for infoIni() to test.
Getters should generally not be logging at debug or trace level since it clutters the log.

In passing move the destructor to the end of the file.
2018-11-16 16:05:06 -05:00
David Steele
6532912d51 Begin v2.08 development. 2018-11-16 10:04:14 -05:00
David Steele
04d9e4d5a8 v2.07: Automatic Backup Checksum Delta
Bug Fixes:

* Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)
* Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
* Fix error after log file open failure when processing should continue. (Reported by vthriller.)

Features:

* Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang.)

Improvements:

* Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)
2018-11-16 09:50:50 -05:00
David Steele
b6f7cbc315 Add destructors to IoRead and IoWrite objects.
These interfaces previously used the memory context of the object they were associated with and did not have their own destructors.

There are times when it is useful to free the interface without also freeing the underlying object so give IoRead and IoWrite their own memory contexts and destructors.

In passing fix a comment type in bufferRead.c.
2018-11-15 16:25:46 -05:00
David Steele
480e1da798 Add ioWriteFlush() to flush pending output.
By default the IoWrite object does not write until the output buffer is full but this is a problem for protocol messages that must be sent in order to get a response.

ioWriteFlush() is not called internally by IoWrite but can be used at any time to immediately write all bytes from the output buffer without closing the IoWrite object.
2018-11-14 08:53:42 -05:00
David Steele
7918629f59 Modify value of PERL_EMBED_ERROR macro.
Remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:16:53 -05:00
David Steele
504e7d77fa Add comment regarding vars being required in blocks.
Documentation block syntax requires that at least one var be specified.

This limitation should be removed but for now add a comment to describe why a bogus var is defined.
2018-11-14 08:13:01 -05:00
David Steele
14d2f8b86c Reword misleading message in stack trace when parameter buffer is full.
The prior message stated that there had been a buffer overrun which is not true since the code prevents that.

In fact, this message means the parameter buffer filled while building the parameter list. Rather than display a partial list we output this message instead.

Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:11:11 -05:00
David Steele
a0b2af567a Add TEST_LOG() and TEST_LOG_FMT() macros.
These macros provide a convenient way to output debug information in tests.

They are not intended to be left in test code when it is committed to master.
2018-11-13 20:58:32 -05:00
David Steele
a582ad837f Add logging macros for TimeMSec type. 2018-11-13 20:43:08 -05:00
David Steele
086bc35ddc Make ioReadLine() read less aggressively.
ioReadLine() calls ioRead(), which aggressively tries to fill the output buffer, but this doesn't play well with blocking reads.

Give ioReadLine() an option that tells it to read only what is available.  That doesn't mean the function will never block but at least it won't do so by reading too far.
2018-11-12 21:18:53 -05:00
David Steele
bc810e5a87 Remove error suppression for pgBackRest make.
This looks like a copy-pasto because make works just fine without it.
2018-11-12 20:44:56 -05:00
David Steele
a2672557f8 Allow storage path and file mode to be 0.
Not all storage drivers support Posix-style permissions, e.g. S3.
2018-11-12 20:00:57 -05:00
David Steele
aff9693e64 Add strEmpty() to String object.
strEmpty() is more concise and explanatory than strSize() == 0.
2018-11-12 14:13:55 -05:00
David Steele
38c5f65770 Use THROW_ON_SYS_ERROR macro to improve fork code coverage.
Lack of code coverage in this module is just noise since these functions cannot generally be made to fail.

Any failures are fatal.
2018-11-11 18:12:43 -05:00
David Steele
72ea47bfb3 Add KernelError to report miscellaneous kernel errors. 2018-11-11 18:07:56 -05:00
David Steele
257df96b12 Add THROW*_ON_SYS_ERROR* macros to test and throw system errors.
These macros check the error result internally and are appropriate for system calls that won't return errors and so break coverage.
2018-11-11 18:06:09 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
3cd43fd60c Add new HTML tags and strExtra to DocHtmlElement.
Add HTML tags for table elements.

The strExtra parameter allows adhoc tags to be added to an element for features that can't be implemented with CSS, e.g. colspan.
2018-11-11 16:34:15 -05:00
David Steele
5e3b7cbe6e Apply String constant macros where appropriate.
This may not be all the places where the macros can be useful but it's a good start.
2018-11-10 18:32:49 -05:00
David Steele
8f857a975e Add constant macros to String object.
There are many places (and the number is growing) where a zero-terminated string constant must be transformed into a String object to be usable.  This pattern wastes time and memory, especially since the created string is generally used in a read-only fashion.

Define macros to create constant String objects that are initialized at compile time rather than at run time.
2018-11-10 09:37:12 -05:00
David Steele
df200bee2a Add regExpPrefix() to aid in static prefix searches.
The storageList() command accepts a regular expression as a filter.  This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code.  However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally.

S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible.

Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever.  It requires a ^ anchor and stops scanning when the first special character is found.
2018-11-09 16:50:22 -05:00
David Steele
8c504bd2f9 Improve alignment of expected vs. got error test results.
It is easier to compare the error messages when they start at the same column.
2018-11-09 10:37:59 -05:00
David Steele
b5a3c8c84b Add strCatChr() to String object. 2018-11-09 09:54:55 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
7cf1f42c5c Add time since the beginning of the run to each test statement.
Output the time in seconds of each test statement since the start of the test run.

This helps find individual tests that are running slowly.
2018-11-08 19:50:31 -05:00
David Steele
2150a26424 Storage interface methods no longer declare the driver as const.
This works for the Posix driver but is bad for more dynamic drivers (e.g. S3) that need write access to the driver object.
2018-11-08 18:03:18 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
a9feaba9e5 Add memContextCallbackClear() to prevent double free() calls.
If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.

Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().
2018-11-07 08:51:32 -05:00