1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1679 Commits

Author SHA1 Message Date
David Steele
c2d0a21d63 Allow secrets to be passed via environment variables.
When environment variables were added in d0b9f986 they were classified as cfgSourceParam, but one of the restrictions on this type is that they can't pass secrets because they might be exposed in the process list.

The solution is to reclassify environment variables as cfgSourceConfig.  This allows them to handle secrets because they will not pass values to subprocesses as parameters.  Instead, each subprocess is expected to check the environment directly during configuration parsing.

In passing, move the error about secrets being passed on the command-line up to command-line parsing and make the error more generic with respect to the configuration file now that multiple configuration files are allowed.
2018-08-30 18:44:40 -04:00
David Steele
70514061fd Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path.
Relative link paths were being combined with the paths of previous links (relative or absolute) due to the $strPath variable being modified in the current iteration rather than simply being passed to the next level of recursion.

This issue did not affect absolute links and relative tablespace links were caught by other checks, though the error was confusing.

Reported by Cynthia Shang.
2018-08-30 16:27:36 -04:00
David Steele
1631f8b3ce Embedded documentation update missed in c6384904. 2018-08-29 16:45:58 -05:00
David Steele
c638490451 Documentation updates for exclude feature based on review.
Reviewed by Cynthia Shang.
2018-08-28 16:49:29 -04:00
David Steele
14cde54b37 Limit manifest build recursion (i.e. links followed) to sixteen levels to detect link loops. 2018-08-28 16:27:10 -04:00
David Steele
a6cecf7d5e Prevent manifest from being built more than once. 2018-08-28 16:22:30 -04:00
Stephen Frost
4498f726a5 Improve asynchronous archive-get/archive-push performance by directly checking status files.
Prior to this commit, an expression was used to search the spool directory for ok/error files for a specific WAL segment. This involved setting up a regular expression and using opendir/readdir.

Instead, directly probe for the status files, checking directly if a '.ok' or '.error' file exists, avoiding the regular expression and eliminating the directory scan.

Only the two files now probed for could have ever matched the regular expression which had been provided and it's unlikely that many more additional files will be added, so this is a good improvement, and optimization, with little downside.

Contributed by Stephen Frost.
2018-08-28 16:04:53 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
David Steele
77dca5b968 Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added. 2018-08-24 19:31:45 -04:00
David Steele
4c3cf435c0 Add cvtZToUInt() to convert string to unsigned int. 2018-08-24 17:08:00 -04:00
Cynthia Shang
eb30d88b6a Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta.
Contributed by Cynthia Shang.
2018-08-24 16:50:33 -04:00
David Steele
0ed37ab9e7 Update Archive::Info->archiveIdList() to return a valid error code instead of unknown. 2018-08-24 12:13:10 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
512da125ad Improve error reporting for TEST_ASSIGN() and TEST_RESULT_VOID() macros. 2018-08-22 19:03:13 -04:00
David Steele
ad989068d2 Fix issue when a boolean option (e.g. delta) was specified more than once.
Reported by Yogesh Sharma.
2018-08-22 16:45:25 -04:00
David Steele
8a8738308c Enable -Wvla. 2018-08-22 14:48:37 -04:00
David Steele
de5614db6b PostgreSQL 11 Beta 3 support. 2018-08-15 14:05:39 -04:00
David Steele
d0b9f986a0 Allow any option to be set in an environment variable.
This includes options that previously could only be specified on the command line, e.g. stanza.
2018-08-15 10:52:53 -04:00
David Steele
4a822d3032 Correct OptionInvalidError to OptionInvalidValueError in boolean option parsing. 2018-08-15 10:32:18 -04:00
David Steele
238e5dae3c Allow resolve array from cd5df357 to autosize to detect short arrays in testing. 2018-08-14 17:17:14 -04:00
David Steele
cb4b715533 Add strReplaceChr() to String object. 2018-08-14 16:49:38 -04:00
David Steele
4a176681c3 Add cvtCharToZ() and macro for debugging char params. 2018-08-14 16:18:17 -04:00
David Steele
9e3273fdf9 Define cipher magic size with sizeof() rather than using a constant. 2018-08-14 16:08:58 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
e3ff6b209d Filters can now produce output that differs from input.
This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 14:21:53 -04:00
David Steele
a45d1b4a60 Update to command-line docs missed in 40924e8f. 2018-08-14 13:38:11 -04:00
David Steele
40924e8f5c Fix invalid log level in log-path option reference.
Reported by Camilo Aguilar.
2018-08-13 07:47:26 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cb6821e853 Stop trying to arrange contributors in release.xml by last/first name.
Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense.

Inspired by Christophe Pettus' PostgresOpen 2017 talk, "Human Beings Do Not Have a Primary Key".
2018-08-12 18:19:34 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
f06bf9e832 Improve error message when a command is missing the stanza option.
Suggested by Sarah Conway.
2018-08-11 09:47:07 -04:00
Cynthia Shang
8ab2e72960 Migrate minimum set of code for reading archive.info files from Perl to C.
Contributed by Cynthia Shang.
2018-08-09 08:57:21 -04:00
David Steele
7993f1a966 Add basic C JSON parser. 2018-08-09 08:06:23 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
ac16cb7ff1 Add cvtBoolToConstZ() to simplify conversion of boolean to string. 2018-07-28 09:39:52 -04:00
David Steele
7a00bbebfa Do nothing in memContextMove() when the context is already in the specified parent. 2018-07-28 09:04:06 -04:00
David Steele
cb975a5330 Remove redundant lines from embedded Perl by combining blank lines. 2018-07-28 08:43:58 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
01aea0c067 Implement filters that do not modify the buffer.
Update cryptoHash to use the new interface.
2018-07-24 21:08:27 -04:00
David Steele
8e1017c162 Move most host setup to containers defined in the documentation.
This includes PostgreSQL installation which had previously been included in the documentation. This way produces faster builds and there is no need for us to document PostgreSQL installation.
2018-07-23 07:59:24 -04:00
David Steele
2723831032 Allow containers to be defined in a document.
The defined containers are built before the document build begins which allows them to be reused.
2018-07-23 07:49:15 -04:00
David Steele
ae72772e5b Fix typo in 18626306. 2018-07-23 07:17:24 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
58e9f1e50c Refactor the common/log tests to not depend on common/harnessLog.
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.

Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
David Steele
1359e2908c Fix issue where errors raised in C were not logged when called from Perl.
pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging.

Reported by Douglas J Hunley.
2018-07-20 08:11:34 -04:00
David Steele
d3cfeebdf9 Rename error-handling variables in Main.pm to conform to standard. 2018-07-20 08:03:44 -04:00