1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

923 Commits

Author SHA1 Message Date
David Steele
df947cfcb2 Add documentation for building the documentation.
A basic primer for building the documentation.  Lots that could be added, but it's a start.
2018-12-12 13:52:23 -05:00
David Steele
fdc76742c8 Use absolute paths so that ./doc.pl runs.
The relative paths used would not correctly find required libraries and DTD files.
2018-12-12 13:48:31 -05:00
David Steele
ee04ebe314 Fix Centos/RHEL 7 documentation builds.
This was caused by a new container version that was released around December 5th.  The new version explicitly denies user logons by leaving /var/run/nologin in place after boot.

The solution is to enable the service that is responsible for removing this file on a successful boot.
2018-12-12 11:15:09 -05:00
Cynthia Shang
2f15a90d18 Add infoArchiveIdHistoryMatch() to the InfoArchive object.
Match a PostgreSQL system identifier and version to a pgBackRest archive id.

Contributed by Cynthia Shang.
2018-12-10 18:45:57 -05:00
Cynthia Shang
f0417ee524 Use cast to make for loop more readable in InfoPg module.
The previous way worked but was a head-scratcher when reading the code.  This cast hopefully makes it a bit more obvious what is going on.

Contributed by Cynthia Shang.
2018-12-10 18:31:49 -05:00
Cynthia Shang
2514d08d0d Remove #define statements in the InfoPg module to conform with newly-adopted coding standards.
Contributed by Cynthia Shang.
2018-12-10 18:15:06 -05:00
Cynthia Shang
fe152b8f29 Rename constants in Info module for consistency.
INFO is generally used as the prefix for info file constants so rename these accordingly.

Also follow newly-adopted coding standards for when #define is required for a static String constant.

Contributed by Cynthia Shang.
2018-12-10 18:08:26 -05:00
Cynthia Shang
e3435ec386 Add path expression for repository backup to the storage helper.
This is the counterpart to the archive path expression and constructs paths into the backup part of the repository.

Contributed by Cynthia Shang.
2018-12-10 17:35:31 -05:00
Cynthia Shang
4f539db8d9 Allow NULL stanza in storage helper.
Some commands (e.g. info) do not take a stanza or the stanza is optional.  In that case it is the job of the command to construct the repository path with a stanza as needed.

Update helper functions to omit the stanza from the constructed path when it is NULL.

Contributed by Cynthia Shang.
2018-12-10 17:01:33 -05:00
Cynthia Shang
cbf514e191 Improve info error messages introduced in 74b72df9.
- Add detail to errors when info files are loaded with incorrect encryption settings.
- Throw FileMissingError rather than FileOpenError when both copies of the info file are missing.
- If one file is present (but errors) and the other is missing, then return the error for the file that was present.

Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
Cynthia Shang
80a3e21521 Add strSizeFormat() to String object.
Converts sizes in bytes to a more human-readable form, .e.g. 1KB, 1.1GB.

Contributed by Cynthia Shang.
2018-12-10 16:11:51 -05:00
David Steele
1c5f8f45b6 Add configuration to the standby so it works as a primary when promoted.
This code was generated during testing and it seemed a good idea to keep it.  It is only a partial solution since the primary also needs additional configuration to be able to fail back and forth.
2018-12-07 12:32:10 -05:00
David Steele
495391c743 Pick pg_switch_wal()/pg_switch_xlog() based on PostgreSQL version.
One step in making the documentation build for all versions that we support.
2018-12-07 12:19:28 -05:00
David Steele
e6abdfb5b8 Add error message when current user uid/gid does not map to a name.
This condition resulted in a nasty stack trace dump when the undefined value was used later on.

Reported by Camilo Aguilar.
2018-12-07 07:41:26 -05:00
David Steele
e73416e9e3 Change file ownership only when required.
Previously chown() would be called even when no ownership changes were required.

In most cases changes are not required and it seems better to perform an extra stat() rather than an extra chown().

Also add unit tests for owner() since there weren't any.
2018-12-05 17:56:47 -05:00
David Steele
e96986a4e1 Error when --target-action=shutdown specified for PostgreSQL < 9.5.
This equaled "promote" on unsupported versions which qualifies as a surprising behavior.
2018-12-05 16:21:45 -05:00
David Steele
5b52f02021 Clarify when target-action is effective and PostgreSQL version support.
Suggested by Keith Fiske.
2018-12-05 15:58:45 -05:00
David Steele
a0ecf53ead Fix documentation typo. 2018-12-05 13:07:55 -05:00
David Steele
bf873be4aa Redact authentication header when throwing S3 errors.
The authentication header contains the access key (not the secret key) so don't include it in errors that can be seen at any log level.

Suggested by Brad Nicholson.
2018-12-05 12:51:13 -05:00
David Steele
cc6447356e Fix test binary name for gprof.
This got missed in 1f8931f7 when the test binary was renamed.

Also output call graph along with the flat report.  The flat report is generally most useful but it doesn't hurt to have both.
2018-12-05 09:15:45 -05:00
David Steele
33fa2ede7d Add Centos/RHEL 7 option to documentation build.
Centos/RHEL 7 builds can be specified with --var=os-type=centos7.

Also added to the documentation release and CI builds.
2018-12-05 07:12:28 -05:00
David Steele
baeff9e4f0 Create common if expressions for testing os-type.
These expressions simplify os-type testing.  This will be especially true as more OS types are added.
2018-12-04 17:33:56 -05:00
David Steele
9e217d0256 Documentation may be built with user-specified packages.
By default the documentation builds pgBackRest from source, but the documentation is also a good way to smoke-test packages.

Allow a package file to be specified by passing --var=package=/path/to/package.ext.  This works for Debian and CentOS 6 builds.
2018-12-04 13:17:55 -05:00
David Steele
0db030fa63 Add coding standards for String constants. 2018-12-04 11:46:25 -05:00
David Steele
14190f9e6c Update URL for Docker install.
As usual the old URL started providing a broken version of Docker rather than producing a clear error message.  This happens once a year or so.
2018-12-03 12:41:53 -05:00
David Steele
17e611cb88 Replace keywords with more flexible if statements.
Keywords were extremely limited and prevented us from generating multi-version documentation and other improvements.

Replace keywords with an if statement that can evaluate a Perl expression with variable replacement.

Since keywords were used to generate cache keys, add a --key-var parameter to identify which variables should make up the key.
2018-12-01 12:40:01 -05:00
David Steele
977506b446 Clarify that region/endpoint must be configured correctly for the bucket.
Suggested by Pritam Barhate.
2018-11-30 11:15:52 -05:00
David Steele
64b97fd7ca Correct archive-get-queue-max to be size type.
This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.

This is not a breaking change because currently configured integer values will be read as bytes.

Reported by Ronan Dunklau.
2018-11-30 10:55:29 -05:00
David Steele
1ad67644da Remove request for S3 object info directly after putting it.
After a file is copied during backup the size is requested from the storage in case it differs from what was written so that repo-size can be reported accurately. This is useful for situations where compression is being done by the filesystem (e.g. ZFS) and what is stored can differ in size from what was written.

In S3 the reported size will always be exactly what was written so there is no need to check the size and doing so immediately can cause problems because the new file might not appear in list commands. This has not been observed on S3 (though it seems to be possible) but it has been reported on the Swift S3 gateway.

Add a driver capability to determine if size needs to be called after a file is written and if not then simply use the number of bytes written for repo-size.

Reported by Matt Kunkel.
2018-11-30 10:38:02 -05:00
David Steele
5d3c8e47f1 Pre-build containers for any execute elements marked pre.
This allows the documentation to be built more quickly and offline during development when --pre is specified on the command line.

Each host gets a pre-built container with all the execute elements marked pre. As long as the pre elements do not change the container will not need to be rebuilt.

The feature should not be used for CI builds as it may hide errors in the documentation.
2018-11-29 14:45:15 -05:00
David Steele
74b72df9db Improve error message when info files are missing/corrupt.
The previous error message only showed the last error.  In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked.

Show both errors below a generic "unable to load" error.  Details are now given explaining exactly why the primary and copy failed.

Previously if one file could not be loaded a warning would be output.  This has been removed because it is not clear what the user should do in this case.  Should they do a stanza-create --force?  Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
David Steele
7c2fcb63e4 Enable encryption for archive-get command in C.
The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.

The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
David Steele
6c23830991 Add encryption capability to Info* objects.
Info objects can be stored encrypted and loaded with a cipher type and passphrase.
2018-11-28 14:41:10 -05:00
David Steele
410a04a58e Allow arbitary InOut filters to be chained in IoFilterGroup.
If InOut filters were placed next to each other then the second filter would never get a NULL input signaling it to flush.  This arrangement only worked if the second filter had some other indication that it should flush, such as a decompression filter where the flush is indicated in the input stream.

This is not a live issue because currently no InOut filters are chained together.
2018-11-28 14:20:12 -05:00
David Steele
838cfa44b7 Allow arbitrary multiplier and flush character in IoTestFilterMultiply.
This allows for more complex test filter chains.

Rename from IoTestFilterDouble to reflect the new functionality.
2018-11-28 14:02:14 -05:00
David Steele
3e254f4cff Add IoFilter interface to CipherBlock object.
This allows CipherBlock to be used as a filter in an IoFilterGroup.  The C-style functions used by Perl are now deprecated and should not be used for any new code.

Also add functions to convert between cipher names and CipherType.
2018-11-28 12:42:36 -05:00
David Steele
c3a84ccae0 Merge cipher.h into crypto.h.
There is not enough in cipher.h to make it worth having a separate header.
2018-11-28 12:03:27 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
Cynthia Shang
f4a1751abc Improve JSON to Variant conversion and add Variant to JSON conversion.
Add boolean and one-dimensional list types to jsonToKv().

Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.

Contributed by Cynthia Shang.
2018-11-23 16:02:33 -05:00
David Steele
e641c130d3 Fix release note typo. 2018-11-23 12:19:56 -05:00
David Steele
beae375330 Enable S3 storage for archive-get command in C.
The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.

The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.
2018-11-23 12:18:07 -05:00
David Steele
b5690e21a4 Allow I/O read interface to explicitly request blocking reads.
TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.

Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.

In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.
2018-11-23 12:01:36 -05:00
David Steele
256b727a3d Add S3 storage driver.
Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-21 19:32:49 -05:00
David Steele
72252ed2a1 Add HttpClient object.
A robust HTTP client with pipelining support and automatic retries.

Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried.

Only the HTTPS protocol is currently supported.
2018-11-21 19:11:45 -05:00
David Steele
1dd06a6e46 Add TlsClient object.
A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.

This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.

Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-21 18:43:25 -05:00
David Steele
ac426bc456 New test containers with static test certificates.
Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.

The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.

Two new certificates have been defined. See test/certificate/README.md for details.

The old dynamic certificates will be retained until they are replaced.
2018-11-21 18:13:37 -05:00