1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1909 Commits

Author SHA1 Message Date
David Steele
ee04ebe314 Fix Centos/RHEL 7 documentation builds.
This was caused by a new container version that was released around December 5th.  The new version explicitly denies user logons by leaving /var/run/nologin in place after boot.

The solution is to enable the service that is responsible for removing this file on a successful boot.
2018-12-12 11:15:09 -05:00
Cynthia Shang
2f15a90d18 Add infoArchiveIdHistoryMatch() to the InfoArchive object.
Match a PostgreSQL system identifier and version to a pgBackRest archive id.

Contributed by Cynthia Shang.
2018-12-10 18:45:57 -05:00
Cynthia Shang
f0417ee524 Use cast to make for loop more readable in InfoPg module.
The previous way worked but was a head-scratcher when reading the code.  This cast hopefully makes it a bit more obvious what is going on.

Contributed by Cynthia Shang.
2018-12-10 18:31:49 -05:00
Cynthia Shang
2514d08d0d Remove #define statements in the InfoPg module to conform with newly-adopted coding standards.
Contributed by Cynthia Shang.
2018-12-10 18:15:06 -05:00
Cynthia Shang
fe152b8f29 Rename constants in Info module for consistency.
INFO is generally used as the prefix for info file constants so rename these accordingly.

Also follow newly-adopted coding standards for when #define is required for a static String constant.

Contributed by Cynthia Shang.
2018-12-10 18:08:26 -05:00
Cynthia Shang
e3435ec386 Add path expression for repository backup to the storage helper.
This is the counterpart to the archive path expression and constructs paths into the backup part of the repository.

Contributed by Cynthia Shang.
2018-12-10 17:35:31 -05:00
Cynthia Shang
4f539db8d9 Allow NULL stanza in storage helper.
Some commands (e.g. info) do not take a stanza or the stanza is optional.  In that case it is the job of the command to construct the repository path with a stanza as needed.

Update helper functions to omit the stanza from the constructed path when it is NULL.

Contributed by Cynthia Shang.
2018-12-10 17:01:33 -05:00
Cynthia Shang
cbf514e191 Improve info error messages introduced in 74b72df9.
- Add detail to errors when info files are loaded with incorrect encryption settings.
- Throw FileMissingError rather than FileOpenError when both copies of the info file are missing.
- If one file is present (but errors) and the other is missing, then return the error for the file that was present.

Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
Cynthia Shang
80a3e21521 Add strSizeFormat() to String object.
Converts sizes in bytes to a more human-readable form, .e.g. 1KB, 1.1GB.

Contributed by Cynthia Shang.
2018-12-10 16:11:51 -05:00
Cynthia Shang
947fa6fc2b Fix comment typo.
Contributed by Cynthia Shang.
2018-12-10 15:43:52 -05:00
David Steele
1c5f8f45b6 Add configuration to the standby so it works as a primary when promoted.
This code was generated during testing and it seemed a good idea to keep it.  It is only a partial solution since the primary also needs additional configuration to be able to fail back and forth.
2018-12-07 12:32:10 -05:00
David Steele
495391c743 Pick pg_switch_wal()/pg_switch_xlog() based on PostgreSQL version.
One step in making the documentation build for all versions that we support.
2018-12-07 12:19:28 -05:00
David Steele
e6abdfb5b8 Add error message when current user uid/gid does not map to a name.
This condition resulted in a nasty stack trace dump when the undefined value was used later on.

Reported by Camilo Aguilar.
2018-12-07 07:41:26 -05:00
Cynthia Shang
ac84ffb901 Clarify that storageExists() is only valid for files.
Contributed by Cynthia Shang.
2018-12-06 11:57:30 -05:00
Cynthia Shang
a72759eb5b Add comments on String and Variant usage regarding memory contexts.
Contributed by Cynthia Shang.
2018-12-06 11:53:38 -05:00
David Steele
11181e69b8 Disable Centos/RHEL 7 documentation builds.
These were introduced in 33fa2ede and ran for a day or so before they started failing consistently on CI.  Local builds work fine.

Disable them to free the pipeline for further commits while we determine the issue.
2018-12-06 09:04:01 -05:00
David Steele
e73416e9e3 Change file ownership only when required.
Previously chown() would be called even when no ownership changes were required.

In most cases changes are not required and it seems better to perform an extra stat() rather than an extra chown().

Also add unit tests for owner() since there weren't any.
2018-12-05 17:56:47 -05:00
David Steele
e96986a4e1 Error when --target-action=shutdown specified for PostgreSQL < 9.5.
This equaled "promote" on unsupported versions which qualifies as a surprising behavior.
2018-12-05 16:21:45 -05:00
David Steele
5b52f02021 Clarify when target-action is effective and PostgreSQL version support.
Suggested by Keith Fiske.
2018-12-05 15:58:45 -05:00
David Steele
a0ecf53ead Fix documentation typo. 2018-12-05 13:07:55 -05:00
David Steele
bf873be4aa Redact authentication header when throwing S3 errors.
The authentication header contains the access key (not the secret key) so don't include it in errors that can be seen at any log level.

Suggested by Brad Nicholson.
2018-12-05 12:51:13 -05:00
David Steele
cc6447356e Fix test binary name for gprof.
This got missed in 1f8931f7 when the test binary was renamed.

Also output call graph along with the flat report.  The flat report is generally most useful but it doesn't hurt to have both.
2018-12-05 09:15:45 -05:00
David Steele
33fa2ede7d Add Centos/RHEL 7 option to documentation build.
Centos/RHEL 7 builds can be specified with --var=os-type=centos7.

Also added to the documentation release and CI builds.
2018-12-05 07:12:28 -05:00
David Steele
baeff9e4f0 Create common if expressions for testing os-type.
These expressions simplify os-type testing.  This will be especially true as more OS types are added.
2018-12-04 17:33:56 -05:00
David Steele
9e217d0256 Documentation may be built with user-specified packages.
By default the documentation builds pgBackRest from source, but the documentation is also a good way to smoke-test packages.

Allow a package file to be specified by passing --var=package=/path/to/package.ext.  This works for Debian and CentOS 6 builds.
2018-12-04 13:17:55 -05:00
David Steele
0db030fa63 Add coding standards for String constants. 2018-12-04 11:46:25 -05:00
David Steele
14190f9e6c Update URL for Docker install.
As usual the old URL started providing a broken version of Docker rather than producing a clear error message.  This happens once a year or so.
2018-12-03 12:41:53 -05:00
David Steele
17e611cb88 Replace keywords with more flexible if statements.
Keywords were extremely limited and prevented us from generating multi-version documentation and other improvements.

Replace keywords with an if statement that can evaluate a Perl expression with variable replacement.

Since keywords were used to generate cache keys, add a --key-var parameter to identify which variables should make up the key.
2018-12-01 12:40:01 -05:00
Cynthia Shang
47490ba960 Fix comment typo.
Contributed by Cynthia Shang.
2018-11-30 18:47:42 -05:00
David Steele
977506b446 Clarify that region/endpoint must be configured correctly for the bucket.
Suggested by Pritam Barhate.
2018-11-30 11:15:52 -05:00
David Steele
64b97fd7ca Correct archive-get-queue-max to be size type.
This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.

This is not a breaking change because currently configured integer values will be read as bytes.

Reported by Ronan Dunklau.
2018-11-30 10:55:29 -05:00
David Steele
1ad67644da Remove request for S3 object info directly after putting it.
After a file is copied during backup the size is requested from the storage in case it differs from what was written so that repo-size can be reported accurately. This is useful for situations where compression is being done by the filesystem (e.g. ZFS) and what is stored can differ in size from what was written.

In S3 the reported size will always be exactly what was written so there is no need to check the size and doing so immediately can cause problems because the new file might not appear in list commands. This has not been observed on S3 (though it seems to be possible) but it has been reported on the Swift S3 gateway.

Add a driver capability to determine if size needs to be called after a file is written and if not then simply use the number of bytes written for repo-size.

Reported by Matt Kunkel.
2018-11-30 10:38:02 -05:00
David Steele
5d3c8e47f1 Pre-build containers for any execute elements marked pre.
This allows the documentation to be built more quickly and offline during development when --pre is specified on the command line.

Each host gets a pre-built container with all the execute elements marked pre. As long as the pre elements do not change the container will not need to be rebuilt.

The feature should not be used for CI builds as it may hide errors in the documentation.
2018-11-29 14:45:15 -05:00
David Steele
74b72df9db Improve error message when info files are missing/corrupt.
The previous error message only showed the last error.  In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked.

Show both errors below a generic "unable to load" error.  Details are now given explaining exactly why the primary and copy failed.

Previously if one file could not be loaded a warning would be output.  This has been removed because it is not clear what the user should do in this case.  Should they do a stanza-create --force?  Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
David Steele
47687dd13a Add xmlNodeAttribute() to XmlNode object.
Retrieves a node attribute unless it is the root node.
2018-11-28 18:10:54 -05:00
David Steele
7c2fcb63e4 Enable encryption for archive-get command in C.
The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.

The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
David Steele
6c23830991 Add encryption capability to Info* objects.
Info objects can be stored encrypted and loaded with a cipher type and passphrase.
2018-11-28 14:41:10 -05:00
David Steele
410a04a58e Allow arbitary InOut filters to be chained in IoFilterGroup.
If InOut filters were placed next to each other then the second filter would never get a NULL input signaling it to flush.  This arrangement only worked if the second filter had some other indication that it should flush, such as a decompression filter where the flush is indicated in the input stream.

This is not a live issue because currently no InOut filters are chained together.
2018-11-28 14:20:12 -05:00
David Steele
838cfa44b7 Allow arbitrary multiplier and flush character in IoTestFilterMultiply.
This allows for more complex test filter chains.

Rename from IoTestFilterDouble to reflect the new functionality.
2018-11-28 14:02:14 -05:00
David Steele
3e254f4cff Add IoFilter interface to CipherBlock object.
This allows CipherBlock to be used as a filter in an IoFilterGroup.  The C-style functions used by Perl are now deprecated and should not be used for any new code.

Also add functions to convert between cipher names and CipherType.
2018-11-28 12:42:36 -05:00
David Steele
c3a84ccae0 Merge cipher.h into crypto.h.
There is not enough in cipher.h to make it worth having a separate header.
2018-11-28 12:03:27 -05:00
David Steele
7df7cf6e6d Rename cipherBlock*() functions to cipherBlock*C().
Some of the old names conflict with the new functions that must be created to implement the filter. Rename these to cipherBlock*C() to indicate that they take C-style parameters.

These functions are only used by the Perl LibC code and will be removed or refactored eventually.
2018-11-27 22:49:24 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
7a1a688835 Add EOF detection to content read in HttpClient.
If the connection closed before all content was sent httpClientRead() would get stuck in an infinite loop waiting for it to arrive.

EOF should never be reached during content read so immediately error if EOF is detected.
2018-11-24 09:12:44 -05:00
Cynthia Shang
f4a1751abc Improve JSON to Variant conversion and add Variant to JSON conversion.
Add boolean and one-dimensional list types to jsonToKv().

Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.

Contributed by Cynthia Shang.
2018-11-23 16:02:33 -05:00
David Steele
e641c130d3 Fix release note typo. 2018-11-23 12:19:56 -05:00