1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

984 Commits

Author SHA1 Message Date
David Steele
be8f31a952 JSON improvements.
Optimize the parser implementation and make the renderer more null tolerant.

Also make some string and variant constructors null tolerant.
2019-02-05 15:34:18 +02:00
David Steele
b29a8dd9c5 Automatically adjust db-timeout when protocol-timeout is smaller.
This already worked in reverse, but this case is needed when a command that only uses protocol-timeout (e.g. info) calls a remote process where protocol-timeout and db-timeout can be set.  If protocol-timeout was set to less than the default db-timeout then an error resulted.
2019-02-02 15:03:19 +02:00
David Steele
abc613b454 Begin v2.10 development. 2019-02-02 14:50:24 +02:00
David Steele
a89a376119 v2.09: Minor Improvements and Bug Fixes
Bug Fixes:

* Fix issue with multiple async status files causing a hard error. (Reported by Vidhya Gurumoorthi, Joe Ayers, Douglas J Hunley.)

Improvements:

* The info command is implemented entirely in C.
* Simplify info command text message when no stanzas are present by replacing the repository path with "the repository".
* Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds. (Contributed by Douglas J Hunley.)
* Update address lookup in C TLS client to use modern methods. (Suggested by Bruno Friedmann.)
* Include Posix-compliant header for strcasecmp() and fd_set. (Suggested by ucando.)
2019-01-30 22:37:35 +02:00
David Steele
aa3e5b8c72 Allow primary gid for the test user to be different from uid.
Apparently up until now they have always been the same, which is pretty typical.  However, if they were not then ContainerTest.pm was not happy.
2019-01-30 17:03:17 +02:00
David Steele
711b3e67cb Add examples for building a documentation host. 2019-01-30 16:53:59 +02:00
David Steele
dada50ee26 Fix hard-coded repository path.
This prevented packages from being passed to the documentation unless they were in the /backrest directory on the host.

Also make the local path /pgbackrest instead of the deprecated /backrest.

Reported by Heath Lord.
2019-01-30 13:39:22 +02:00
David Steele
8634eb8c43 Include Posix-compliant header for fd_set.
Similar to f3ae3c4f, add Posix-compliant header to make FreeBSD happy.

Suggested by ucando.
2019-01-29 12:08:14 +02:00
David Steele
b5a103f2df Improve P/PP type macro handling.
Rather than create _P/_PP variants for every type that needs to pass/return pointers, create FUNCTION_*_P/PP() macros that will properly pass or return any single/double pointer types.

There remain a few unresolved edge cases such as CHARPY but this handles the majority of types well.
2019-01-28 22:33:29 +02:00
David Steele
8b2a344cfe Remove unused type parameter from FUNCTION_TEST_RETURN().
This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well.

The original intention was probably to allow logging of TEST return values but that never happened.
2019-01-28 15:06:28 +02:00
David Steele
7333b6302f Improve CONST type macro handling.
Rather than create a CONST_ variant for every type that needs to be returned const, create a FUNCTION_LOG_RETURN_CONST() macro that will return any type as const.
2019-01-28 09:44:10 +02:00
David Steele
d29aa61286 Allocate extra space for concatenations in the String object.
The string object was reallocating memory with every concatenation which is not very efficient.  This is especially true for JSON rendering which does a lot of concatenations.

Instead allocate a pool of extra memory on the first concatenation (50% of size) to be used for future concatenations and reallocate when needed.

Also add a 1GB size limit to ensure that there are no overflows.
2019-01-27 11:50:09 +02:00
David Steele
82c2d615b3 Move MACRO_TO_STR() to common/debug.h.
This macro is handy for constructing static message strings so move it where the core code can see it.
2019-01-27 11:34:12 +02:00
David Steele
4027123ef1 Update contributor name. 2019-01-27 08:01:17 +02:00
David Steele
8f6d324b2c Fix issue with multiple async status files causing a hard error.
Multiple status files were being created by asynchronous archiving if a high-level error occurred after one or more WAL segments had already been transferred successfully.  Error files were being written for every file in the queue regardless of whether it had already succeeded.  To fix this, add an option to skip writing error files when an ok file already exists.

There are other situations where both files might exist (various fsync and filesystem error scenarios) so it seems best to retry in the case that multiple status files are found rather than throwing a hard error (which then means that archiving is completely stuck).  In the case of multiple status files, a warning will be logged to alert the user that something unusual is happening and the command will be retried.

Reported by fpa-postgres, Joe Ayers, Douglas J Hunley.
2019-01-26 16:59:54 +02:00
David Steele
f3ae3c4f9d Include Posix-compliant header for strcasecmp().
gcc has apparently merged this function in string.h but Posix specifies that it should be in strings.h.  FreeBSD at at least is sticking to the standard.

In the long run it might be better to implement our own strcasecmp() function but for now just add the header.

Suggested by ucando.
2019-01-26 13:48:46 +02:00
David Steele
1401c023f0 Fix release note typo. 2019-01-26 12:02:48 +02:00
David Steele
053972bfe0 Update address lookup in C TLS client to use modern methods.
The implementation using gethostbyname() was only intended to be used during prototyping but was forgotten when the code was finalized.

Replace it with gettaddrinfo() which is more modern and supports IPv6.

Suggested by Bruno Friedmann.
2019-01-26 12:01:18 +02:00
David Steele
4d1060ea6b Replace FileOpenError with HostConnectError in TlsClient.
HostConnectError is more appropriate in this context.
2019-01-26 10:39:51 +02:00
David Steele
33d39d248c Allow if condition in documentation lists and list items. 2019-01-24 16:45:28 +02:00
David Steele
bec52b6f41 Allow if in manifest variables.
The code supported this feature but the manifest DTD did now allow it.
2019-01-24 11:11:12 +02:00
David Steele
db08656537 Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency.
Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do.  Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging.  Leave FUNCTION_TEST_* macros as they are.

Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds.  It was difficult to figure out when an assert would be checked with all the different types in play.  When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
David Steele
d245f8eb42 The info command is implemented entirely in C.
The C info code has already been committed but this commit wires it into main.

Also remove the info Perl code and tests since they are no longer called.
2019-01-21 13:51:45 +02:00
David Steele
f79af47bd4 Increase timeout in storage/s3 module to improve reliability. 2019-01-19 20:44:40 +02:00
David Steele
e57f1b5293 Fix escaping in JSON string parser. 2019-01-19 12:57:40 +02:00
David Steele
7355248d6b Add remote storage objects.
This is a partial implementation of remote storage with just enough functionality to get the info command working.  The client is written in C but the server is still in Perl, which limits progress until a C server is written.
2019-01-18 22:04:37 +02:00
David Steele
88201f37a3 Add ProtocolClient object and helper functions.
This is a complete protocol client implementation in C.

Currently there is no C server implementation so the C client is talking to a Perl server.  This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure.  While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run.

Just as in Perl there are helper functions to make constructing protocol objects easier.  Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
David Steele
9cac403f61 Add Exec object.
Executes a child process and allows the calling process to communicate with it using read/write io.

This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose
execution.
2019-01-18 11:45:40 +02:00
David Steele
797f8098d1 Add ioReadBuf() to easily read into a buffer.
Moves some boilerplate into a function and makes it easier to get coverage in cases where a single buffer read captures all the data.
2019-01-18 11:14:44 +02:00
David Steele
06d41b4dc0 Add cfgExecParam() to generate parameters for executing commands.
Parameters for the local/remote commands are based on parameters that are passed to the current command.

Generate parameters for the new command based on the intersection of parameters between the current command and the command to be executed.
2019-01-17 22:29:19 +02:00
David Steele
ecd56105e6 Add IoHandleRead and IoHandleWrite objects.
General i/o objects for reading and writing file descriptors, in particular those that can block.  In other words, these are not generally to be used with file descriptors for actual files, but rather pipes, sockets, etc.
2019-01-17 22:08:31 +02:00
David Steele
bf0c41d9d6 Add const VariantList * debug type. 2019-01-16 22:23:25 +02:00
David Steele
e68d1e7304 Simplify info command text message when no stanza are present.
Replace the repository path with just "the repository".  The path is not important in this context and it is clearer to state where the stanzas are missing from.
2019-01-16 19:23:10 +02:00
David Steele
ef9dc89e08 Update Storage::Local->list() to accept an undefined path.
The Perl code has a tendency to generate absolute paths even when they are not needed. This change helps the C and Perl storage work together via the protocol layer.
2019-01-16 18:49:12 +02:00
David Steele
b4146b6bff Update Perl repo rules to work when stanza is not specified.
The C storage object strives to use rules whenever possible instead of generating absolute paths.  This change helps the C and Perl storage work together via the protocol layer.
2019-01-16 18:45:19 +02:00
David Steele
0014e15944 Fix escaping in JSON string renderer. 2019-01-16 18:34:50 +02:00
David Steele
c0a9048536 Fix null output in JSON renderer.
In some cases nulls were being rendered as "(null)" because no special logic existed to handle them.
2019-01-16 09:11:01 +02:00
David Steele
ec26a0594e Allow string Variant objects to contain null.
This is more useful than the variant itself being null as it saves conditionals when creating.
2019-01-16 09:05:13 +02:00
David Steele
1b334da87f Add kvMove() and varLstMove().
Allow these objects to be moved to another mem context.
2019-01-14 22:02:23 +02:00
David Steele
aab9e38b9a Return UnknownError from errorTypeFromCode() for invalid error codes.
The prior behavior was to throw an exception but this was not very helpful when something unexpected happened.  Better to at least emit the error message even if the error code is not very helpful.
2019-01-14 21:34:22 +02:00
David Steele
2b02d37602 Allow empty arrays in JSON parser. 2019-01-14 21:21:37 +02:00
David Steele
8304d452b3 Make the C version of the info command conform to the Perl version.
There were some small differences in ordering and how the C version handled missing directories.  It may be that the C version is more consistent, but for now it is more important to be compatible with the Perl version.

These differences were missed because the C info command was not wired into main.c so it was not being tested in regression.  This commit does not fix the wiring issue because there will likely be a release soon and it is too big a change to put in at the last moment.
2019-01-13 22:44:58 +02:00
David Steele
f314a1f8aa Improve accuracy of strSizeFormat().
Casting to int caused large values to be slightly inaccurate so cast to uint64_t instead.

Also, use multiplication where possible since the compiler should precompute multiplied values.
2019-01-13 22:22:15 +02:00
David Steele
cebcb3f027 Null-terminate list returned by strLstPtr().
The null-terminator is required by many C functions that accept a string list so just add it in all cases rather than making it optional.
2019-01-10 09:49:28 +02:00
David Steele
ade2c3102d Ignore SIGPIPE signals and check EPIPE result instead.
SIGPIPE immediately terminates the process but we would rather catch the EPIPE error and gracefully shutdown.

Ignore SIGPIPE and throw the EPIPE error via normal error handling.
2019-01-06 17:42:44 +02:00
Douglas J Hunley
7272d6e247 Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds.
For some reason adding -D_POSIX_C_SOURCE=200112L caused MacOS builds to stop working.  Combining both flags seems to work fine for all tested systems.

Contributed by Douglas J Hunley.
2019-01-06 17:28:17 +02:00
David Steele
9560baf659 Move C module include in test.c above headers included for testing.
Including the C module after the headers required for testing meant that if headers were missing from the C module they were not caught while directly testing the C module.

The missing headers were caught in general testing, but it is frustrating to get an error in a module that has already passed while testing another module or running CI.

Move the C module include to the very top so missing headers cause immediate failures.
2019-01-06 15:52:59 +02:00
David Steele
1de22cac2b Rename common/io/handle module to common/io/handleWrite.
ioHandleWriteOneStr() will become a helper function for the IoHandleWrite object.
2019-01-06 14:37:39 +02:00
David Steele
200fd3aa05 Clarify that encryption is always performed client-side.
Suggested by Bruce Burdick.
2019-01-04 12:25:58 +02:00
David Steele
50717aa846 Begin v2.09 development. 2019-01-04 11:00:59 +02:00
David Steele
db24ff8df4 v2.08: Minor Improvements and Bug Fixes
Bug Fixes:

* Remove request for S3 object info directly after putting it. (Reported by Matt Kunkel.)
* Correct archive-get-queue-max to be size type. (Reported by Ronan Dunklau.)
* Add error message when current user uid/gid does not map to a name. (Reported by Camilo Aguilar.)
* Error when --target-action=shutdown specified for PostgreSQL < 9.5.

Improvements:

* Set TCP keepalives on S3 connections. (Suggested by Ronan Dunklau.)
* Reorder info command text output so most recent backup is output last. (Contributed by Cynthia Shang. Suggested by Ryan Lambert.)
* Change file ownership only when required.
* Redact authentication header when throwing S3 errors. (Suggested by Brad Nicholson.)
2019-01-02 22:04:47 +02:00
David Steele
26c888873e Merge common/typeVariantListTest module into common/typeVariantTest.
These modules are closely related so it makes sense for them to be merged.
2019-01-01 18:14:43 +02:00
David Steele
07b9176f25 Merge common/typeStringListTest module into common/typeStringTest.
These modules are closely related so it makes sense for them to be merged.
2019-01-01 18:05:13 +02:00
David Steele
493fb78787 Add strLstInsert() and strLstInsertZ() to StringList object.
Use lstInsert() to implement insertions into any position in a StringList.
2019-01-01 17:31:23 +02:00
Cynthia Shang
cc6376fdb6 Modify general document elements to allow any child element.
This allows for nesting of elements such as <b> and <i>.

Contributed by Cynthia Shang.
2018-12-31 09:19:38 +02:00
Cynthia Shang
2b7440ddf3 Allow custom logo for PDF documentation.
Contributed by Cynthia Shang.
2018-12-31 08:51:00 +02:00
Cynthia Shang
72865ca33b Add admonitions to documentation renderers.
Admonitions call out places where the user should take special care.

Support added for HTML, PDF, Markdown and help text renderers.  XML files have been updated accordingly.

Contributed by Cynthia Shang.
2018-12-30 16:40:20 +02:00
Cynthia Shang
3dc327fd05 Add optional table captions.
All renderers now support table captions, when requested.

Contributed by Cynthia Shang.
2018-12-20 23:20:54 +02:00
Cynthia Shang
8cf8d29f12 Escape special characters in latex when not in a code block.
A number of common characters are not allowed in latex without being escaped.

Also convert some HTML-specific codes that are used in the documentation.

Contributed by Cynthia Shang.
2018-12-19 19:13:26 +02:00
David Steele
23b583336f Set TCP keepalives on S3 connections.
Keepalives may help in situations where RST packets are being blocked by a firewall or otherwise do not arrive.

The C code uses select on all reads so it should never block, but add keepalives just in case.

Suggested by Ronan Dunklau.
2018-12-18 22:12:59 +02:00
Cynthia Shang
96028073cb Add HTML table rendering and update PDF/Markdown renderers to support header-less tables.
Contributed by Cynthia Shang.
2018-12-18 22:02:23 +02:00
Cynthia Shang
b85e51d6d5 Base menu ordering on natural ordering in the manifest.
Menu ordering was alphabetical which is not ideal for a user interface.

Contributed by Cynthia Shang.
2018-12-14 18:46:12 -05:00
Cynthia Shang
35bbb5bd68 Reorder info command text output so most recent backup is output last.
After a stanza-upgrade backups for the old cluster are displayed until they expire.  Cluster info was output newest to oldest which meant after an upgrade the most recent backup would no longer be output last.

Update the text output ordering so the most recent backup is always output last.

Contributed by Cynthia Shang.
Suggested by Ryan Lambert.
2018-12-14 18:25:31 -05:00
Cynthia Shang
205525b607 Migrate local info command to C.
The info command will only be executed in C if the repository is local, i.e. not located on a remote repository host.  S3 is considered "local" in this case.

This is a direct migration from Perl to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.

Contributed by Cynthia Shang.
2018-12-13 16:22:34 -05:00
Cynthia Shang
e6ef40e8a3 Add infoBackup object to encapsulate the backup.info file.
The infoBackup object is the counterpart to the infoArchive object which encapsulates the archive.info file.

Currently the object is read-only, i.e. it is not possible to create a new or modify an existing backup.info file.

There a number of constants that will also be used in the infoManifest object so go ahead and create a module to contain them so they don't need to be moved later.

Contributed by Cynthia Shang.
2018-12-13 15:46:18 -05:00
David Steele
d6dfbfd2f3 Add zlib1g-dev to Debian builds.
Not all debian-based distros were getting this installed.
2018-12-12 14:35:44 -05:00
David Steele
df947cfcb2 Add documentation for building the documentation.
A basic primer for building the documentation.  Lots that could be added, but it's a start.
2018-12-12 13:52:23 -05:00
David Steele
fdc76742c8 Use absolute paths so that ./doc.pl runs.
The relative paths used would not correctly find required libraries and DTD files.
2018-12-12 13:48:31 -05:00
Cynthia Shang
2f15a90d18 Add infoArchiveIdHistoryMatch() to the InfoArchive object.
Match a PostgreSQL system identifier and version to a pgBackRest archive id.

Contributed by Cynthia Shang.
2018-12-10 18:45:57 -05:00
Cynthia Shang
f0417ee524 Use cast to make for loop more readable in InfoPg module.
The previous way worked but was a head-scratcher when reading the code.  This cast hopefully makes it a bit more obvious what is going on.

Contributed by Cynthia Shang.
2018-12-10 18:31:49 -05:00
Cynthia Shang
2514d08d0d Remove #define statements in the InfoPg module to conform with newly-adopted coding standards.
Contributed by Cynthia Shang.
2018-12-10 18:15:06 -05:00
Cynthia Shang
fe152b8f29 Rename constants in Info module for consistency.
INFO is generally used as the prefix for info file constants so rename these accordingly.

Also follow newly-adopted coding standards for when #define is required for a static String constant.

Contributed by Cynthia Shang.
2018-12-10 18:08:26 -05:00
Cynthia Shang
e3435ec386 Add path expression for repository backup to the storage helper.
This is the counterpart to the archive path expression and constructs paths into the backup part of the repository.

Contributed by Cynthia Shang.
2018-12-10 17:35:31 -05:00
Cynthia Shang
4f539db8d9 Allow NULL stanza in storage helper.
Some commands (e.g. info) do not take a stanza or the stanza is optional.  In that case it is the job of the command to construct the repository path with a stanza as needed.

Update helper functions to omit the stanza from the constructed path when it is NULL.

Contributed by Cynthia Shang.
2018-12-10 17:01:33 -05:00
Cynthia Shang
cbf514e191 Improve info error messages introduced in 74b72df9.
- Add detail to errors when info files are loaded with incorrect encryption settings.
- Throw FileMissingError rather than FileOpenError when both copies of the info file are missing.
- If one file is present (but errors) and the other is missing, then return the error for the file that was present.

Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
Cynthia Shang
80a3e21521 Add strSizeFormat() to String object.
Converts sizes in bytes to a more human-readable form, .e.g. 1KB, 1.1GB.

Contributed by Cynthia Shang.
2018-12-10 16:11:51 -05:00
David Steele
1c5f8f45b6 Add configuration to the standby so it works as a primary when promoted.
This code was generated during testing and it seemed a good idea to keep it.  It is only a partial solution since the primary also needs additional configuration to be able to fail back and forth.
2018-12-07 12:32:10 -05:00
David Steele
495391c743 Pick pg_switch_wal()/pg_switch_xlog() based on PostgreSQL version.
One step in making the documentation build for all versions that we support.
2018-12-07 12:19:28 -05:00
David Steele
e6abdfb5b8 Add error message when current user uid/gid does not map to a name.
This condition resulted in a nasty stack trace dump when the undefined value was used later on.

Reported by Camilo Aguilar.
2018-12-07 07:41:26 -05:00
David Steele
e73416e9e3 Change file ownership only when required.
Previously chown() would be called even when no ownership changes were required.

In most cases changes are not required and it seems better to perform an extra stat() rather than an extra chown().

Also add unit tests for owner() since there weren't any.
2018-12-05 17:56:47 -05:00
David Steele
e96986a4e1 Error when --target-action=shutdown specified for PostgreSQL < 9.5.
This equaled "promote" on unsupported versions which qualifies as a surprising behavior.
2018-12-05 16:21:45 -05:00
David Steele
5b52f02021 Clarify when target-action is effective and PostgreSQL version support.
Suggested by Keith Fiske.
2018-12-05 15:58:45 -05:00
David Steele
a0ecf53ead Fix documentation typo. 2018-12-05 13:07:55 -05:00
David Steele
bf873be4aa Redact authentication header when throwing S3 errors.
The authentication header contains the access key (not the secret key) so don't include it in errors that can be seen at any log level.

Suggested by Brad Nicholson.
2018-12-05 12:51:13 -05:00
David Steele
cc6447356e Fix test binary name for gprof.
This got missed in 1f8931f7 when the test binary was renamed.

Also output call graph along with the flat report.  The flat report is generally most useful but it doesn't hurt to have both.
2018-12-05 09:15:45 -05:00
David Steele
33fa2ede7d Add Centos/RHEL 7 option to documentation build.
Centos/RHEL 7 builds can be specified with --var=os-type=centos7.

Also added to the documentation release and CI builds.
2018-12-05 07:12:28 -05:00
David Steele
baeff9e4f0 Create common if expressions for testing os-type.
These expressions simplify os-type testing.  This will be especially true as more OS types are added.
2018-12-04 17:33:56 -05:00
David Steele
9e217d0256 Documentation may be built with user-specified packages.
By default the documentation builds pgBackRest from source, but the documentation is also a good way to smoke-test packages.

Allow a package file to be specified by passing --var=package=/path/to/package.ext.  This works for Debian and CentOS 6 builds.
2018-12-04 13:17:55 -05:00
David Steele
0db030fa63 Add coding standards for String constants. 2018-12-04 11:46:25 -05:00
David Steele
14190f9e6c Update URL for Docker install.
As usual the old URL started providing a broken version of Docker rather than producing a clear error message.  This happens once a year or so.
2018-12-03 12:41:53 -05:00
David Steele
17e611cb88 Replace keywords with more flexible if statements.
Keywords were extremely limited and prevented us from generating multi-version documentation and other improvements.

Replace keywords with an if statement that can evaluate a Perl expression with variable replacement.

Since keywords were used to generate cache keys, add a --key-var parameter to identify which variables should make up the key.
2018-12-01 12:40:01 -05:00
David Steele
977506b446 Clarify that region/endpoint must be configured correctly for the bucket.
Suggested by Pritam Barhate.
2018-11-30 11:15:52 -05:00
David Steele
64b97fd7ca Correct archive-get-queue-max to be size type.
This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.

This is not a breaking change because currently configured integer values will be read as bytes.

Reported by Ronan Dunklau.
2018-11-30 10:55:29 -05:00
David Steele
1ad67644da Remove request for S3 object info directly after putting it.
After a file is copied during backup the size is requested from the storage in case it differs from what was written so that repo-size can be reported accurately. This is useful for situations where compression is being done by the filesystem (e.g. ZFS) and what is stored can differ in size from what was written.

In S3 the reported size will always be exactly what was written so there is no need to check the size and doing so immediately can cause problems because the new file might not appear in list commands. This has not been observed on S3 (though it seems to be possible) but it has been reported on the Swift S3 gateway.

Add a driver capability to determine if size needs to be called after a file is written and if not then simply use the number of bytes written for repo-size.

Reported by Matt Kunkel.
2018-11-30 10:38:02 -05:00
David Steele
5d3c8e47f1 Pre-build containers for any execute elements marked pre.
This allows the documentation to be built more quickly and offline during development when --pre is specified on the command line.

Each host gets a pre-built container with all the execute elements marked pre. As long as the pre elements do not change the container will not need to be rebuilt.

The feature should not be used for CI builds as it may hide errors in the documentation.
2018-11-29 14:45:15 -05:00
David Steele
74b72df9db Improve error message when info files are missing/corrupt.
The previous error message only showed the last error.  In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked.

Show both errors below a generic "unable to load" error.  Details are now given explaining exactly why the primary and copy failed.

Previously if one file could not be loaded a warning would be output.  This has been removed because it is not clear what the user should do in this case.  Should they do a stanza-create --force?  Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
David Steele
7c2fcb63e4 Enable encryption for archive-get command in C.
The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.

The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
David Steele
6c23830991 Add encryption capability to Info* objects.
Info objects can be stored encrypted and loaded with a cipher type and passphrase.
2018-11-28 14:41:10 -05:00
David Steele
410a04a58e Allow arbitary InOut filters to be chained in IoFilterGroup.
If InOut filters were placed next to each other then the second filter would never get a NULL input signaling it to flush.  This arrangement only worked if the second filter had some other indication that it should flush, such as a decompression filter where the flush is indicated in the input stream.

This is not a live issue because currently no InOut filters are chained together.
2018-11-28 14:20:12 -05:00
David Steele
838cfa44b7 Allow arbitrary multiplier and flush character in IoTestFilterMultiply.
This allows for more complex test filter chains.

Rename from IoTestFilterDouble to reflect the new functionality.
2018-11-28 14:02:14 -05:00
David Steele
3e254f4cff Add IoFilter interface to CipherBlock object.
This allows CipherBlock to be used as a filter in an IoFilterGroup.  The C-style functions used by Perl are now deprecated and should not be used for any new code.

Also add functions to convert between cipher names and CipherType.
2018-11-28 12:42:36 -05:00
David Steele
c3a84ccae0 Merge cipher.h into crypto.h.
There is not enough in cipher.h to make it worth having a separate header.
2018-11-28 12:03:27 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
Cynthia Shang
f4a1751abc Improve JSON to Variant conversion and add Variant to JSON conversion.
Add boolean and one-dimensional list types to jsonToKv().

Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.

Contributed by Cynthia Shang.
2018-11-23 16:02:33 -05:00
David Steele
e641c130d3 Fix release note typo. 2018-11-23 12:19:56 -05:00
David Steele
beae375330 Enable S3 storage for archive-get command in C.
The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.

The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.
2018-11-23 12:18:07 -05:00
David Steele
b5690e21a4 Allow I/O read interface to explicitly request blocking reads.
TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.

Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.

In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.
2018-11-23 12:01:36 -05:00
David Steele
256b727a3d Add S3 storage driver.
Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-21 19:32:49 -05:00
David Steele
72252ed2a1 Add HttpClient object.
A robust HTTP client with pipelining support and automatic retries.

Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried.

Only the HTTPS protocol is currently supported.
2018-11-21 19:11:45 -05:00
David Steele
1dd06a6e46 Add TlsClient object.
A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.

This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.

Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-21 18:43:25 -05:00
David Steele
ac426bc456 New test containers with static test certificates.
Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.

The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.

Two new certificates have been defined. See test/certificate/README.md for details.

The old dynamic certificates will be retained until they are replaced.
2018-11-21 18:13:37 -05:00
David Steele
53e3651cca Remove embedded semicolon from String constant macros.
The embedded semicolon led to inconsistent semicolons when using the macro and is not our general convention.

Remove embedded semicolons from the macros and add semicolons in usage where they were not present.
2018-11-21 08:30:14 -05:00
David Steele
bc25db5667 Add interface objects for libxml2.
Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.

This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-20 20:40:11 -05:00
David Steele
6680130c6f Require S3 key options except for local/remote commands.
S3 key options (repo1-s3-key/repo1-s3-key-secret) were not required which meant that users got an ugly assertion when they were missing rather than a tidy configuration error.

Only the local/remote commands need them to be optional.  This is because local/remote commands get all their options from the command line but secrets cannot be passed on the command line.  Instead, secrets are passed to the local/remote commands via the protocol for any operation that needs them.

The configuration system allows required to be set per command so use that to improve the error messages while not breaking the local/remote commands.
2018-11-20 19:24:53 -05:00
David Steele
f743d4e924 Add testRepoPath() to let C unit tests know where the code repository is located.
This allows a C unit test to access data in the code repository that might be useful for testing.

Add testRepoPathSet() to set the repository path.

In passing remove extra whitespace in the TEST_RESULT_VOID() macro.
2018-11-20 15:48:56 -05:00
David Steele
b0659278cc Add ServiceError for errors from a service that can be retried.
An example is HTTP 5xx errors which should mostly be retried.
2018-11-16 17:22:22 -05:00
David Steele
3aa4fda573 Add missing LOG_DEBUG() macro. 2018-11-16 16:17:24 -05:00
David Steele
d0e553e4c2 Reduce debug level for infoIni() to test.
Getters should generally not be logging at debug or trace level since it clutters the log.

In passing move the destructor to the end of the file.
2018-11-16 16:05:06 -05:00
David Steele
6532912d51 Begin v2.08 development. 2018-11-16 10:04:14 -05:00
David Steele
04d9e4d5a8 v2.07: Automatic Backup Checksum Delta
Bug Fixes:

* Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)
* Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
* Fix error after log file open failure when processing should continue. (Reported by vthriller.)

Features:

* Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang.)

Improvements:

* Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)
2018-11-16 09:50:50 -05:00
David Steele
b6f7cbc315 Add destructors to IoRead and IoWrite objects.
These interfaces previously used the memory context of the object they were associated with and did not have their own destructors.

There are times when it is useful to free the interface without also freeing the underlying object so give IoRead and IoWrite their own memory contexts and destructors.

In passing fix a comment type in bufferRead.c.
2018-11-15 16:25:46 -05:00
David Steele
480e1da798 Add ioWriteFlush() to flush pending output.
By default the IoWrite object does not write until the output buffer is full but this is a problem for protocol messages that must be sent in order to get a response.

ioWriteFlush() is not called internally by IoWrite but can be used at any time to immediately write all bytes from the output buffer without closing the IoWrite object.
2018-11-14 08:53:42 -05:00
David Steele
7918629f59 Modify value of PERL_EMBED_ERROR macro.
Remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:16:53 -05:00
David Steele
14d2f8b86c Reword misleading message in stack trace when parameter buffer is full.
The prior message stated that there had been a buffer overrun which is not true since the code prevents that.

In fact, this message means the parameter buffer filled while building the parameter list. Rather than display a partial list we output this message instead.

Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:11:11 -05:00
David Steele
a0b2af567a Add TEST_LOG() and TEST_LOG_FMT() macros.
These macros provide a convenient way to output debug information in tests.

They are not intended to be left in test code when it is committed to master.
2018-11-13 20:58:32 -05:00
David Steele
a582ad837f Add logging macros for TimeMSec type. 2018-11-13 20:43:08 -05:00
David Steele
086bc35ddc Make ioReadLine() read less aggressively.
ioReadLine() calls ioRead(), which aggressively tries to fill the output buffer, but this doesn't play well with blocking reads.

Give ioReadLine() an option that tells it to read only what is available.  That doesn't mean the function will never block but at least it won't do so by reading too far.
2018-11-12 21:18:53 -05:00
David Steele
bc810e5a87 Remove error suppression for pgBackRest make.
This looks like a copy-pasto because make works just fine without it.
2018-11-12 20:44:56 -05:00
David Steele
a2672557f8 Allow storage path and file mode to be 0.
Not all storage drivers support Posix-style permissions, e.g. S3.
2018-11-12 20:00:57 -05:00
David Steele
aff9693e64 Add strEmpty() to String object.
strEmpty() is more concise and explanatory than strSize() == 0.
2018-11-12 14:13:55 -05:00
David Steele
38c5f65770 Use THROW_ON_SYS_ERROR macro to improve fork code coverage.
Lack of code coverage in this module is just noise since these functions cannot generally be made to fail.

Any failures are fatal.
2018-11-11 18:12:43 -05:00
David Steele
72ea47bfb3 Add KernelError to report miscellaneous kernel errors. 2018-11-11 18:07:56 -05:00
David Steele
257df96b12 Add THROW*_ON_SYS_ERROR* macros to test and throw system errors.
These macros check the error result internally and are appropriate for system calls that won't return errors and so break coverage.
2018-11-11 18:06:09 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
3cd43fd60c Add new HTML tags and strExtra to DocHtmlElement.
Add HTML tags for table elements.

The strExtra parameter allows adhoc tags to be added to an element for features that can't be implemented with CSS, e.g. colspan.
2018-11-11 16:34:15 -05:00
David Steele
5e3b7cbe6e Apply String constant macros where appropriate.
This may not be all the places where the macros can be useful but it's a good start.
2018-11-10 18:32:49 -05:00
David Steele
8f857a975e Add constant macros to String object.
There are many places (and the number is growing) where a zero-terminated string constant must be transformed into a String object to be usable.  This pattern wastes time and memory, especially since the created string is generally used in a read-only fashion.

Define macros to create constant String objects that are initialized at compile time rather than at run time.
2018-11-10 09:37:12 -05:00
David Steele
df200bee2a Add regExpPrefix() to aid in static prefix searches.
The storageList() command accepts a regular expression as a filter.  This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code.  However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally.

S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible.

Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever.  It requires a ^ anchor and stops scanning when the first special character is found.
2018-11-09 16:50:22 -05:00
David Steele
8c504bd2f9 Improve alignment of expected vs. got error test results.
It is easier to compare the error messages when they start at the same column.
2018-11-09 10:37:59 -05:00
David Steele
b5a3c8c84b Add strCatChr() to String object. 2018-11-09 09:54:55 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
7cf1f42c5c Add time since the beginning of the run to each test statement.
Output the time in seconds of each test statement since the start of the test run.

This helps find individual tests that are running slowly.
2018-11-08 19:50:31 -05:00
David Steele
2150a26424 Storage interface methods no longer declare the driver as const.
This works for the Posix driver but is bad for more dynamic drivers (e.g. S3) that need write access to the driver object.
2018-11-08 18:03:18 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
a9feaba9e5 Add memContextCallbackClear() to prevent double free() calls.
If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.

Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().
2018-11-07 08:51:32 -05:00