1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

792 Commits

Author SHA1 Message Date
David Steele
4a176681c3 Add cvtCharToZ() and macro for debugging char params. 2018-08-14 16:18:17 -04:00
David Steele
9e3273fdf9 Define cipher magic size with sizeof() rather than using a constant. 2018-08-14 16:08:58 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
e3ff6b209d Filters can now produce output that differs from input.
This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 14:21:53 -04:00
David Steele
40924e8f5c Fix invalid log level in log-path option reference.
Reported by Camilo Aguilar.
2018-08-13 07:47:26 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cb6821e853 Stop trying to arrange contributors in release.xml by last/first name.
Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense.

Inspired by Christophe Pettus' PostgresOpen 2017 talk, "Human Beings Do Not Have a Primary Key".
2018-08-12 18:19:34 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
f06bf9e832 Improve error message when a command is missing the stanza option.
Suggested by Sarah Conway.
2018-08-11 09:47:07 -04:00
Cynthia Shang
8ab2e72960 Migrate minimum set of code for reading archive.info files from Perl to C.
Contributed by Cynthia Shang.
2018-08-09 08:57:21 -04:00
David Steele
7993f1a966 Add basic C JSON parser. 2018-08-09 08:06:23 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
ac16cb7ff1 Add cvtBoolToConstZ() to simplify conversion of boolean to string. 2018-07-28 09:39:52 -04:00
David Steele
7a00bbebfa Do nothing in memContextMove() when the context is already in the specified parent. 2018-07-28 09:04:06 -04:00
David Steele
cb975a5330 Remove redundant lines from embedded Perl by combining blank lines. 2018-07-28 08:43:58 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
01aea0c067 Implement filters that do not modify the buffer.
Update cryptoHash to use the new interface.
2018-07-24 21:08:27 -04:00
David Steele
8e1017c162 Move most host setup to containers defined in the documentation.
This includes PostgreSQL installation which had previously been included in the documentation. This way produces faster builds and there is no need for us to document PostgreSQL installation.
2018-07-23 07:59:24 -04:00
David Steele
2723831032 Allow containers to be defined in a document.
The defined containers are built before the document build begins which allows them to be reused.
2018-07-23 07:49:15 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
58e9f1e50c Refactor the common/log tests to not depend on common/harnessLog.
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.

Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
David Steele
1359e2908c Fix issue where errors raised in C were not logged when called from Perl.
pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging.

Reported by Douglas J Hunley.
2018-07-20 08:11:34 -04:00
David Steele
d3cfeebdf9 Rename error-handling variables in Main.pm to conform to standard. 2018-07-20 08:03:44 -04:00
David Steele
fa53e2eddb Show exact log level required for stack trace param output instead of just "debug". 2018-07-20 07:15:28 -04:00
David Steele
0ac176b722 Abstract IO layer out of the storage layer.
This allows the routines to be used for IO objects that do not have a storage representation.

Implement buffer read and write IO objects.
2018-07-19 16:04:20 -04:00
David Steele
5dc8a2ec08 storageFileRead() accepts a buffer for output rather than creating one.
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
2018-07-17 19:01:54 -04:00
Cynthia Shang
0acf705416 Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
Contributed by Cynthia Shang.
2018-07-16 17:25:15 -04:00
Cynthia Shang
4e38cbaea9 Add iniSectionList() to Ini object and remove dead code.
Contributed by Cynthia Shang.
2018-07-12 15:28:46 -04:00
Cynthia Shang
0e6b927a17 Add uint64 variant type and supporting conversion functions.
Contributed by Cynthia Shang.
Reviewed by Stephen Frost.
2018-07-12 15:23:18 -04:00
David Steele
0e331b12ba Improve performance of string to int conversion.
Use strtoll() instead of sprintf() for conversion. Also use available integer min/max constants rather than hard-coded values.

Reviewed by Stephen Frost.
Suggested by Stephen Frost.
2018-07-12 15:15:32 -04:00
David Steele
cf889790e6 Error if LibC build is performed outside test environment.
LibC is no longer required for production builds.
2018-07-10 15:39:03 -04:00
David Steele
b1bc53657d Begin v2.05 development. 2018-07-09 08:15:16 -04:00
David Steele
a8143ec125 v2.04: Critical Bug Fix for Backup Resume
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.

Bug Fixes:

* Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
* Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reported by Nj Baliyan.)
* Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz.)

Features:

* PostgreSQL 11 Beta 2 support.

Improvements:

* Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
* Set search_path = 'pg_catalog' on PostgreSQL connections. (Suggested by Stephen Frost.)
2018-07-05 20:16:41 -04:00
David Steele
6f6dad6dd5 Remove call to lscpu which can vary widely by build host. 2018-07-05 16:49:46 -04:00
David Steele
22adb395db Add zero-length file to mock/all test. 2018-07-05 15:40:50 -04:00
David Steele
db17973cd0 Fix critical bug in resume that resulted in inconsistent backups.
A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".

Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 14:01:57 -04:00
David Steele
3d07e01fe9 Make ls ordering deterministic in mock/all test. 2018-07-01 20:02:37 -04:00
Andrew Schwartz
1bd98b61df Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.
AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.

Fixed by Andrew Schwartz.
2018-07-01 08:17:27 -04:00
David Steele
7e65ddad34 PostgreSQL 11 Beta 2 support. 2018-06-30 14:55:25 -04:00
David Steele
7b0e65d488 Improve the HTTP client to set content-length to 0 when not specified by the server.
S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.

Suggested by Adam K. Sumner.
2018-06-26 17:27:22 -04:00
David Steele
e8c68b9c66 Update parameters for VBoxService start. 2018-06-26 16:56:05 -04:00
David Steele
ead70e16bb Add sample S3 policy to restrict bucket privileges.
Suggested by Douglas J Hunley, Jason O'Donnell.
2018-06-23 13:09:56 -04:00
David Steele
aa41e00c9c Set search_path = 'pg_catalog' on PostgreSQL connections.
Suggested by Stephen Frost.
2018-06-21 11:39:37 -04:00
David Steele
7fee739710 Remove Debian package patch since it has been committed upstream. 2018-06-20 18:26:53 -04:00
David Steele
6084e9be04 Fix default location of pgbackrest.conf in option reference. 2018-06-16 13:07:01 -04:00
David Steele
1a0d568600 Preliminary documentation for PostgreSQL 11 unprivileged user backup. 2018-06-14 19:05:35 -04:00
David Steele
3793ae1e4f Convert the not very portable uint type to unsigned int.
Suggested by Devrim Gündüz.
2018-06-14 17:59:10 -04:00
Cynthia Shang
4e7692b74d Fix error in selective restore when only one user database exists in the cluster.
Fixed by Cynthia Shang.
Reported by Nj Baliyan.
2018-06-14 17:44:44 -04:00
David Steele
fbd1da0523 Create a new section to describe building pgBackRest and build on a separate host. 2018-06-13 13:43:17 -04:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
064ec757e9 Rename cipher module to the more general crypto. 2018-06-11 10:53:16 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
5c0099781d Update Debian package to add debug symbols to pgBackRest executable. 2018-06-06 22:19:01 -04:00
David Steele
a385cb520b Update primary test environment (Vagrant and Docker) to Ubuntu 18.04. 2018-06-06 15:52:28 -04:00
David Steele
d309a85b51 PostgreSQL 11 Beta 1 support. 2018-06-05 08:59:17 -04:00
David Steele
e57840d739 Document generator improvements.
* Allow parameters to be passed when a container is created.
* Allow /etc/hosts update to be skipped (for containers without bash).
2018-05-25 13:42:09 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
915b09635a Remove RHEL package patch since it has been committed upstream. 2018-05-24 11:00:47 -04:00
David Steele
8eb9921c7c Auto-generate Makefile with dependencies. 2018-05-23 14:57:08 -04:00
David Steele
2f6017511f Split log levels into separate header file.
Many modules that use debug.h do not need to do logging so this reduces dependencies for those modules.
2018-05-23 09:45:08 -04:00
David Steele
dfa92b359d Begin v2.04 development. 2018-05-23 09:10:14 -04:00
David Steele
f7123c1bbf v2.03: Single Executable to Deploy
Bug Fixes:

* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)

Improvements:

* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
David Steele
e69d5fbf62 Fix potential buffer overrun in error message handling.
Reported by Lætitia.
2018-05-21 08:06:31 -04:00
David Steele
72a9684d4e Print total runtime in milliseconds at command end. 2018-05-18 12:03:39 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c Test harness improvements.
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
bce3d0fe1c Build libc using links rather than referencing the C files in src directly.
The C library builds with different options which should not be reused for the C binary or vice versa.
2018-05-17 17:09:06 -04:00
David Steele
91be372e6a Set log-timestamp=n for integration tests.
This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 11:24:38 -04:00
David Steele
a1be8e16cd Update SSL error message test on CentOS 7. 2018-05-11 10:39:36 -04:00
David Steele
4b4af1cbfc Use time_t instead of __time_t for better portability.
Suggested by Nick Floersch.
2018-05-09 11:14:35 -04:00
David Steele
a9069f3fd3 Begin v2.03 development. 2018-05-09 10:50:05 -04:00
David Steele
ee8aafb3ca v2.02: Parallel Asynchronous Archive Get and Configuration Includes
Bug Fixes:

* Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Fix archive-copy throwing "path not found" error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
* Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
* Fix delta restore failing when a linked file is missing.
* Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)

Features:

* Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
* Add support for additional pgBackRest configuration files in the directory specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang.)
* Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma.)

Improvements:

* Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang.)
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
* Show command parameters as well as command options in initial info log message.
* Rename archive-queue-max option to archive-push-queue-max to avoid confusion with the new archive-get-queue-max option. The old option name will continue to be accepted.
2018-05-06 19:53:42 -04:00
David Steele
4d6a51ac47 Add HARNESS_FORK for tests that require fork().
A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
David Steele
790f7c7312 Enforce full branch coverage in C code. 2018-05-05 11:50:11 -04:00
David Steele
0a860e0b60 Full branch coverage for command/help/help, common/error, common/ini, and common/log modules. 2018-05-05 09:38:09 -04:00
David Steele
c3a8fbe706 Add _FMT variants for all THROW macros so format types are checked by the compiler. 2018-05-03 11:24:29 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
5e090ba305 Fix failure in manifest build when two or more files in PGDATA are linked to the same directory.
Reported by Vitaliy Kukharik.
2018-05-02 12:19:54 -04:00
David Steele
6a01db2b32 Fix delta restore failing when a linked file was missing. 2018-05-02 12:06:12 -04:00
David Steele
9d9fad88a2 Add example of command section usage to archiving configuration.
Suggested by Christophe Courtois.
2018-05-02 07:31:44 -04:00
David Steele
fd1984239e Fix rendering of key/value and list options in help.
Reported by Clinton Adams.
2018-05-01 14:07:08 -04:00
David Steele
8dbbf2de92 Fix archive-copy throwing "path not found" error for incr/diff backups.
Reported by yummyliu, Vitaliy Kukharik.
2018-05-01 12:37:06 -04:00
David Steele
71ba08f579 Use path list in the backup manifest to do restore path syncs.
Remove recursive path sync functionality since it is no longer used.
2018-05-01 11:05:37 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
c48b0a2a1e Update docs with 32-bit support and caveats.
32-bit support was added in v1.26.

Reported by Viorel Tabara.
2018-04-30 08:13:56 -04:00
David Steele
321a28f6b0 Add walSegmentNext() and walSegmentRange(). 2018-04-29 11:47:50 -04:00
David Steele
be02c67503 Add pgControlInfo() to read pg_control and determine the PostgreSQL version. 2018-04-29 11:20:51 -04:00
David Steele
d74c167420 Add bufNewC() and only reallocate buffer when the size has changed. 2018-04-29 10:41:05 -04:00
David Steele
d44848baa0 Add strLstExists(), strLstExistsZ(), strSub(), and strSubN() to String and StringList objects. 2018-04-29 10:32:46 -04:00
Cynthia Shang
721432f6ae Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024.
Contributed by Cynthia Shang.
2018-04-25 15:46:05 -04:00
Cynthia Shang
8eabc52f3b Add strUpper, strLower, strChr, and strTrunc to String object.
Contributed by Cynthia Shang.
2018-04-25 15:16:42 -04:00
David Steele
88a633da17 Divide tests into three types (unit, integration, performance).
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.

Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 09:12:25 -04:00
David Steele
bb8c315cad Storage object improvements.
* Add storageCopy(), storageMove(), and storagePathSync().
* Separate StorageFile object into separate read and write objects.
* Abstract out Posix file read/write objects.
2018-04-23 17:26:27 -04:00
David Steele
02cc8ccbd4 Add THROWP_* macro variants for error handling.
These macros allow an ErrorType pointer to be passed and are required for functions that may return different errors based on a parameter.
2018-04-23 10:24:04 -04:00
David Steele
18882cb882 Add bufEq and bufCat to Buffer object. 2018-04-23 10:19:45 -04:00
David Steele
f146502eae Add TEST_ERROR_FMT macro to simplify testing of formatted error messages. 2018-04-23 10:10:55 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
ab26c765d4 Don't copy CFGDEF_NAME_ALT or CFGDEF_INHERIT when processing config option inheritance. 2018-04-18 18:57:45 -04:00
David Steele
ea4046a607 Try tweaking time sync settings to prevent clock drift rather than restarting VBoxService on every test run. 2018-04-18 18:47:43 -04:00
David Steele
0381945caa Show command parameters as well as command options in initial info message. 2018-04-17 18:47:14 -04:00
David Steele
e9790fd3b1 Update out-of-date description for the spool-path option. 2018-04-17 17:38:25 -04:00
David Steele
cf0ec44ede Rename Perl tests so they don't conflict with their C counterparts. 2018-04-17 12:03:03 -04:00
David Steele
dce11ce3cf Generate code counts for all source files.
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00
David Steele
3e22977484 Use pip 9.03 in test VMs.
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
2018-04-16 15:25:50 -04:00
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier.
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
David Steele
49fc737cd0 Add monitoring examples using PostgreSQL and jq.
Suggested by Stephen Frost, Brian Faherty.
2018-04-13 14:31:33 -04:00
David Steele
d924f3d50a Add logic to find the real oid of the test1 database during restore testing. 2018-04-13 12:00:22 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
4744eb9387 Add storagePathRemove() and use it in the Perl Posix driver.
This implementation should be faster because it does not stat each file.  It simply assumes that most directory entries are files so attempts an unlink() first.  If the entry is reported by error codes to be a directory then it attempts an rmdir().
2018-04-11 08:21:09 -04:00
David Steele
c9ce20d41a Skip version checks when testing in --dev mode. 2018-04-09 16:46:36 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module.
These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all.
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 12:16:32 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml.
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 10:19:24 -04:00
David Steele
b77b425096 Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave. 2018-04-07 19:44:00 -04:00
David Steele
82751b3b51 Split debug and assert code into separate headers.
Assert can be used earlier because it only depends on the error-handler and not logging.
2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad More C type cleanup.
* Remove typec.h entirely.
* Order all typdefs above local includes.
2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3 Allow MemContext objects to be copied to a new parent.
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4 Cleanup C types.
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
2018-04-05 22:38:11 -04:00
David Steele
bd25248df0 Abstract Posix storage driver code into a separate module. 2018-04-05 14:48:16 -04:00
David Steele
348278bb68 Make backup directory sync more efficient.
Scanning the entire backup directory can be very expensive if there are a lot of small tables. The backup manifest contains the backup directory list so use it to perform syncs instead of scanning the backup directory.
2018-04-03 21:30:15 -04:00
David Steele
0dce8ba2fa Make backup.history sync more efficient.
Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.
2018-04-03 20:54:18 -04:00
David Steele
5890272247 Fix directory syncs running recursively when only the specified directory should be synced.
Reported by Craig A. James.
2018-04-03 18:12:03 -04:00
David Steele
46b6be6d72 Fix header exclusion defines that do not match the general pattern. 2018-04-03 13:35:56 -04:00
David Steele
93fdb98d15 Storage object improvements.
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
David Steele
90f980fe91 Remove documentation describing info --output=json as experimental. 2018-03-30 21:33:30 -04:00
David Steele
635caff573 Add ASSERT() that is preserved in production builds. 2018-03-30 19:10:34 -04:00
David Steele
a8721ffe11 Improve logic for smart builds to include version changes. 2018-03-27 22:48:22 -04:00
David Steele
c1ab7b3c98 Start work on C handle io object and use it to output help. 2018-03-25 11:39:24 -04:00
David Steele
9001b9b957 Replace THROW_ON_SYS_ERROR() with THROW_SYS_ERROR().
The former macro was hiding missing branch coverage for critical error handling.
2018-03-24 14:11:29 -04:00
David Steele
c7a6e9d2f2 Document build improvements.
* Perform apt-get update to ensure packages are up to date before installing.
* Add -p to the repository mkdir so it won't fail if the directory already exists, handy for testing packages.
2018-03-23 16:08:32 -04:00
David Steele
dabf727d91 Include VM type in gcov path to avoid conflicts between VMs with different architectures. 2018-03-21 16:08:08 -04:00
David Steele
c6457b070f Full branch coverage for config/parse module. 2018-03-21 13:46:08 -04:00
David Steele
b234f43c9d Refactor usec to msec in common/time.c.
The implementation provides usec resolution but this is not needed in practice and it makes the interface more complicated due to the extra zeros.
2018-03-21 09:18:48 -04:00
David Steele
07f38f584a Use lcov for C unit test coverage reporting.
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.

Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
2018-03-19 23:33:28 -04:00
David Steele
2a3d6ecde8 v2.01: Minor Bug Fixes and Improvements
Bug Fixes:

* Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
* Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line. Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem. Update command documentation to omit secure options that cannot be specified on the command-line. (Reported by Brad Nicholson.)
* Fix issue passing --no-config to embedded Perl. (Reported by Ibrahim Edib Kokdemir.)
* Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected. The max value for a local/remote process is now error since there is no reason for these processes to emit warnings. (Reported by Clinton Adams.)
* Fix manifest test in the check command when tablespaces are present. (Fixed by Cynthia Shang. Reported by Thomas Flatley.)

Improvements:

* Error when multiple arguments are set in the config file for an option that does not accept multiple arguments. (Contributed by Cynthia Shang.)
* Remove extraneous sudo commands from src/Makefile. (Contributed by Adrian Vondendriesch.)
2018-03-19 21:49:41 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present.
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
2018-03-19 19:54:03 -04:00
David Steele
8e9e97728c Remove --smart from --expect tests.
This ensures that new binaries are built before running the tests.
2018-03-19 17:38:44 -04:00
David Steele
2c1687721e Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected.
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.

Reported by Clinton Adams.
2018-03-19 13:08:42 -04:00
David Steele
0c313713b1 Use clang for static code analysis during lint testing.
Nothing found except for some functions that should have been marked __noreturn__.
2018-03-18 13:32:19 -04:00
David Steele
f0451c1494 Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings. 2018-03-15 13:53:03 -04:00
David Steele
8a1ce42c30 Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line.
Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem.

Update command documentation to omit secure options that cannot be specified on the command-line.

Reported by Brad Nicholson.
2018-03-15 12:02:09 -04:00
David Steele
dd3ce70810 Config test code writes secure options to a file instead of passing on the command-line. 2018-03-15 11:09:46 -04:00
David Steele
4fb75c9cc1 Improve conversion of C exceptions to Exception objects.
Colons in the message would prevent all of the message from being loaded into the Exception object.
2018-03-15 11:03:28 -04:00
David Steele
599e291b36 Check int size in common/type.h.
This ensures that integers are at least 32-bits without having to run the test suite.
2018-03-14 14:34:06 -04:00
David Steele
e7fd94ee23 Simplify table of contents on command page by only listing commands.
Suggested by Stephen Frost.
2018-03-14 11:10:46 -04:00
David Steele
4b6cd92a4c Improve debugging.
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a Add ASSERT_DEBUG() macro for debugging.
Replace all current assert() calls except in tests that can't use the debug code.
2018-03-12 14:31:22 -04:00
Cynthia Shang
dd31ae832d Port most of Config::Config::configLoad() from Perl to C.
Contributed by Cynthia Shang.
2018-03-12 11:25:20 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
94494a632f Update pip before installing awscli. 2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
b987f05f35 Improve bin and libc build performance. 2018-03-09 21:24:30 -05:00
David Steele
c32ee34612 Build with -DNDEBUG by default but disable for testing. 2018-03-09 18:28:34 -05:00
David Steele
fbe4c40386 Move logic for setting Perl configuration defaults to C. 2018-03-09 09:30:50 -05:00
David Steele
919635b6f4 Set config before Main::main() call to avoid secrets being exposed in a stack trace. 2018-03-08 16:24:16 -05:00
Cynthia Shang
34898ce471 Error when multiple arguments are set in the config file for an option that does not accept multiple arguments.
Contributed by Cynthia Shang.
2018-03-08 14:15:05 -05:00
David Steele
e331564c90 Add new test for Common::Io::Process.
Show that output on stderr will raise an exception on close() even if the exit code is 0.
2018-03-08 12:36:55 -05:00
David Steele
62a9bdbd4c Fix issue passing --no-config to embedded Perl.
Reported by Ibrahim Edib Kokdemir.
2018-03-08 10:43:09 -05:00
David Steele
5e0ce3e996 Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate.
Reported by Brad Nicholson.
2018-03-06 17:12:16 -05:00
David Steele
4c8ed3530c Show index in examples for indexed options, i.e. repo-*, pg-*.
Suggested by Stephen Frost.
2018-03-06 14:20:12 -05:00
Cynthia Shang
45ffe37e4f Add cfgOptionTest().
Update cfgOption() calls that are better implemented as cfgOptionTest().

Contributed by Cynthia Shang.
2018-03-05 18:51:48 -05:00
David Steele
a50b1eb5be Remove references to the C library being optional. 2018-03-05 10:03:10 -05:00
David Steele
c07aa2ae19 Improve code documentation in config module. 2018-03-02 22:42:02 -05:00
David Steele
406acf9cba Enable -Wconversion and silence new warnings. 2018-03-02 12:07:12 -05:00
Adrian Vondendriesch
6ca6424fb1 Remove extraneous sudo commands from src/Makefile.
Contributed by Adrian Vondendriesch.
2018-03-01 16:59:49 -05:00
David Steele
af805af044 Remove Debian package patch now that it has been merged upstream. 2018-03-01 16:59:13 -05:00
David Steele
5d1722b8ef Disable console display of coverage for C files.
Devel::Cover does not handle it well.
2018-03-01 13:32:51 -05:00
David Steele
5b491b3035 Add CentOS/RHEL package builds. 2018-02-26 19:32:27 -05:00
David Steele
f9c1a3b156 Enable -Wswitch-enum and silence new warnings. 2018-02-26 10:43:49 -05:00
David Steele
c8df690931 Fix incorrect enum types in config.c that throw warnings under clang.
Found by Douglas J Hunley.
2018-02-26 10:02:33 -05:00
David Steele
13cdb75ac5 v2.00: Performance Improvements for Archive Push
Features:

* The archive-push command is now partially coded in C which allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process. (Reviewed by Cynthia Shang.)

Improvements:

* Improve check command to verify that the backup manifest can be built. (Contributed by Cynthia Shang.)
* Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
* Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
* Allow any non-command-line option to be reset to default on the command-line. This allows options in pgbackrest.conf to be reset to default which reduces the need to write new configuration files for specific needs.
* The C library is now required. This eliminates conditional loading and eases development of new library features.
* The pgbackrest executable is now a C binary instead of Perl. This allows certain time-critical commands (like async archive-push) to run more quickly.
* Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. repo-* options are now indexed although currently only one is allowed.
2018-02-23 22:38:45 -05:00
David Steele
ed5876810d Fix missing variable replace for PDF code-block title.
Found by Jason O'Donnell.
2018-02-23 14:16:20 -05:00
David Steele
2485fabbad PDF rendering improvements.
* Fill is now the default for table columns.
* Column width is specified as a percentage rather that using latex-specific notation.
2018-02-22 09:31:53 -05:00
David Steele
5eb682a569 Add id param for hosts created with host-add.
The host-*-ip variable is created from the id param so the name param can be changed without affecting the host-*-ip variable. If id is not specified then it is copied from name.
2018-02-21 18:10:53 -05:00
David Steele
246d441507 More flexible replacements for PDF titles and footers. 2018-02-21 15:30:35 -05:00
David Steele
aed578a4bd Allow sections to be excluded from table of contents in PDF output. 2018-02-21 15:26:30 -05:00
David Steele
599e41a251 Improve S3 delete performance.
The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
2018-02-18 14:54:32 -05:00
David Steele
9070325760 Improve performance of HTTPS client.
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
2018-02-18 14:53:27 -05:00
David Steele
3eed4bbb43 PDF rendering improvements.
* Check both doc-path and bin-path for logo.
* Allow PDF to be output to a location other than the output directory.
* Use PDF-specific version variable for more flexible formatting.
2018-02-16 15:09:37 -05:00
Cynthia Shang
0e27060952 Add table render for Markdown format.
Contributed by Cynthia Shang.
2018-02-16 14:55:48 -05:00
David Steele
ca52a0485e Allow code blocks to have a type.
Currently this is only rendered in Markdown.
2018-02-16 14:46:02 -05:00
David Steele
2dc0737e95 Improve section source feature to not require a title or content.
The title will be pulled from the source document.
2018-02-16 14:39:42 -05:00
David Steele
1f62237684 Clarify that S3 buckets must be created by the user.
Suggested by David Youatt.
2018-02-14 19:52:46 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp().
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
81952c41f4 Report coverage errors via the console.
This helps with debugging coverage issues on remote services like Travis.
2018-02-14 17:13:18 -05:00
David Steele
a907fd7d2d Build C binary/library for C unit tests that require Perl. 2018-02-14 10:33:21 -05:00
David Steele
213724907b Add contributor. 2018-02-09 22:27:21 -05:00
David Steele
a3b80768bd Merge 1.28 release notes. 2018-02-09 15:56:22 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml.
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5 Update URL for Debian package repository. 2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00