1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

1682 Commits

Author SHA1 Message Date
David Steele
e3ff6b209d Filters can now produce output that differs from input.
This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 14:21:53 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
f06bf9e832 Improve error message when a command is missing the stanza option.
Suggested by Sarah Conway.
2018-08-11 09:47:07 -04:00
Cynthia Shang
8ab2e72960 Migrate minimum set of code for reading archive.info files from Perl to C.
Contributed by Cynthia Shang.
2018-08-09 08:57:21 -04:00
David Steele
7993f1a966 Add basic C JSON parser. 2018-08-09 08:06:23 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
7a00bbebfa Do nothing in memContextMove() when the context is already in the specified parent. 2018-07-28 09:04:06 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
01aea0c067 Implement filters that do not modify the buffer.
Update cryptoHash to use the new interface.
2018-07-24 21:08:27 -04:00
David Steele
ae72772e5b Fix typo in 18626306. 2018-07-23 07:17:24 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
58e9f1e50c Refactor the common/log tests to not depend on common/harnessLog.
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.

Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
David Steele
1359e2908c Fix issue where errors raised in C were not logged when called from Perl.
pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging.

Reported by Douglas J Hunley.
2018-07-20 08:11:34 -04:00
David Steele
d3cfeebdf9 Rename error-handling variables in Main.pm to conform to standard. 2018-07-20 08:03:44 -04:00
David Steele
fa53e2eddb Show exact log level required for stack trace param output instead of just "debug". 2018-07-20 07:15:28 -04:00
David Steele
0ac176b722 Abstract IO layer out of the storage layer.
This allows the routines to be used for IO objects that do not have a storage representation.

Implement buffer read and write IO objects.
2018-07-19 16:04:20 -04:00
David Steele
5dc8a2ec08 storageFileRead() accepts a buffer for output rather than creating one.
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
2018-07-17 19:01:54 -04:00
Cynthia Shang
0acf705416 Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
Contributed by Cynthia Shang.
2018-07-16 17:25:15 -04:00
Cynthia Shang
4e38cbaea9 Add iniSectionList() to Ini object and remove dead code.
Contributed by Cynthia Shang.
2018-07-12 15:28:46 -04:00
Cynthia Shang
0e6b927a17 Add uint64 variant type and supporting conversion functions.
Contributed by Cynthia Shang.
Reviewed by Stephen Frost.
2018-07-12 15:23:18 -04:00
David Steele
0e331b12ba Improve performance of string to int conversion.
Use strtoll() instead of sprintf() for conversion. Also use available integer min/max constants rather than hard-coded values.

Reviewed by Stephen Frost.
Suggested by Stephen Frost.
2018-07-12 15:15:32 -04:00
David Steele
cf889790e6 Error if LibC build is performed outside test environment.
LibC is no longer required for production builds.
2018-07-10 15:39:03 -04:00
David Steele
22adb395db Add zero-length file to mock/all test. 2018-07-05 15:40:50 -04:00
David Steele
db17973cd0 Fix critical bug in resume that resulted in inconsistent backups.
A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".

Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 14:01:57 -04:00
David Steele
3d07e01fe9 Make ls ordering deterministic in mock/all test. 2018-07-01 20:02:37 -04:00
Andrew Schwartz
1bd98b61df Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.
AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.

Fixed by Andrew Schwartz.
2018-07-01 08:17:27 -04:00
David Steele
7e65ddad34 PostgreSQL 11 Beta 2 support. 2018-06-30 14:55:25 -04:00
David Steele
7b0e65d488 Improve the HTTP client to set content-length to 0 when not specified by the server.
S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.

Suggested by Adam K. Sumner.
2018-06-26 17:27:22 -04:00
David Steele
e8c68b9c66 Update parameters for VBoxService start. 2018-06-26 16:56:05 -04:00
David Steele
7fee739710 Remove Debian package patch since it has been committed upstream. 2018-06-20 18:26:53 -04:00
David Steele
3793ae1e4f Convert the not very portable uint type to unsigned int.
Suggested by Devrim Gündüz.
2018-06-14 17:59:10 -04:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
064ec757e9 Rename cipher module to the more general crypto. 2018-06-11 10:53:16 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
5c0099781d Update Debian package to add debug symbols to pgBackRest executable. 2018-06-06 22:19:01 -04:00
David Steele
a385cb520b Update primary test environment (Vagrant and Docker) to Ubuntu 18.04. 2018-06-06 15:52:28 -04:00
David Steele
d309a85b51 PostgreSQL 11 Beta 1 support. 2018-06-05 08:59:17 -04:00
David Steele
e57840d739 Document generator improvements.
* Allow parameters to be passed when a container is created.
* Allow /etc/hosts update to be skipped (for containers without bash).
2018-05-25 13:42:09 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
915b09635a Remove RHEL package patch since it has been committed upstream. 2018-05-24 11:00:47 -04:00
David Steele
8eb9921c7c Auto-generate Makefile with dependencies. 2018-05-23 14:57:08 -04:00
David Steele
2f6017511f Split log levels into separate header file.
Many modules that use debug.h do not need to do logging so this reduces dependencies for those modules.
2018-05-23 09:45:08 -04:00
David Steele
f7123c1bbf v2.03: Single Executable to Deploy
Bug Fixes:

* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)

Improvements:

* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
David Steele
e69d5fbf62 Fix potential buffer overrun in error message handling.
Reported by Lætitia.
2018-05-21 08:06:31 -04:00
David Steele
72a9684d4e Print total runtime in milliseconds at command end. 2018-05-18 12:03:39 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c Test harness improvements.
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
91be372e6a Set log-timestamp=n for integration tests.
This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 11:24:38 -04:00
David Steele
a1be8e16cd Update SSL error message test on CentOS 7. 2018-05-11 10:39:36 -04:00
David Steele
ee8aafb3ca v2.02: Parallel Asynchronous Archive Get and Configuration Includes
Bug Fixes:

* Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Fix archive-copy throwing "path not found" error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
* Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
* Fix delta restore failing when a linked file is missing.
* Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)

Features:

* Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
* Add support for additional pgBackRest configuration files in the directory specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang.)
* Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma.)

Improvements:

* Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang.)
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
* Show command parameters as well as command options in initial info log message.
* Rename archive-queue-max option to archive-push-queue-max to avoid confusion with the new archive-get-queue-max option. The old option name will continue to be accepted.
2018-05-06 19:53:42 -04:00
David Steele
4d6a51ac47 Add HARNESS_FORK for tests that require fork().
A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
David Steele
790f7c7312 Enforce full branch coverage in C code. 2018-05-05 11:50:11 -04:00
David Steele
0a860e0b60 Full branch coverage for command/help/help, common/error, common/ini, and common/log modules. 2018-05-05 09:38:09 -04:00
David Steele
90aadc6534 Full branch coverage for config module. 2018-05-04 12:49:25 -04:00
David Steele
6c3b40152f Improve module branch coverage.
Add full branch coverage for cipher/block, command/command, common/wait, and postgres/pageChecksum modules.
2018-05-04 09:21:32 -04:00
David Steele
c3a8fbe706 Add _FMT variants for all THROW macros so format types are checked by the compiler. 2018-05-03 11:24:29 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
5e090ba305 Fix failure in manifest build when two or more files in PGDATA are linked to the same directory.
Reported by Vitaliy Kukharik.
2018-05-02 12:19:54 -04:00
David Steele
fd1984239e Fix rendering of key/value and list options in help.
Reported by Clinton Adams.
2018-05-01 14:07:08 -04:00
David Steele
1a1ed8d6b9 Update help test to use restore command to provide better coverage. 2018-05-01 13:20:48 -04:00
David Steele
8dbbf2de92 Fix archive-copy throwing "path not found" error for incr/diff backups.
Reported by yummyliu, Vitaliy Kukharik.
2018-05-01 12:37:06 -04:00
David Steele
71ba08f579 Use path list in the backup manifest to do restore path syncs.
Remove recursive path sync functionality since it is no longer used.
2018-05-01 11:05:37 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
321a28f6b0 Add walSegmentNext() and walSegmentRange(). 2018-04-29 11:47:50 -04:00
David Steele
be02c67503 Add pgControlInfo() to read pg_control and determine the PostgreSQL version. 2018-04-29 11:20:51 -04:00
David Steele
8c6e2bdbc7 Add storageInfo() and track size in read objects. 2018-04-29 11:02:21 -04:00
David Steele
d74c167420 Add bufNewC() and only reallocate buffer when the size has changed. 2018-04-29 10:41:05 -04:00
David Steele
d44848baa0 Add strLstExists(), strLstExistsZ(), strSub(), and strSubN() to String and StringList objects. 2018-04-29 10:32:46 -04:00
David Steele
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
Cynthia Shang
721432f6ae Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024.
Contributed by Cynthia Shang.
2018-04-25 15:46:05 -04:00
Cynthia Shang
8eabc52f3b Add strUpper, strLower, strChr, and strTrunc to String object.
Contributed by Cynthia Shang.
2018-04-25 15:16:42 -04:00
David Steele
88a633da17 Divide tests into three types (unit, integration, performance).
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.

Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 09:12:25 -04:00
David Steele
bb8c315cad Storage object improvements.
* Add storageCopy(), storageMove(), and storagePathSync().
* Separate StorageFile object into separate read and write objects.
* Abstract out Posix file read/write objects.
2018-04-23 17:26:27 -04:00
David Steele
18882cb882 Add bufEq and bufCat to Buffer object. 2018-04-23 10:19:45 -04:00
David Steele
f146502eae Add TEST_ERROR_FMT macro to simplify testing of formatted error messages. 2018-04-23 10:10:55 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
ea4046a607 Try tweaking time sync settings to prevent clock drift rather than restarting VBoxService on every test run. 2018-04-18 18:47:43 -04:00
David Steele
0381945caa Show command parameters as well as command options in initial info message. 2018-04-17 18:47:14 -04:00
David Steele
e9790fd3b1 Update out-of-date description for the spool-path option. 2018-04-17 17:38:25 -04:00
David Steele
cf0ec44ede Rename Perl tests so they don't conflict with their C counterparts. 2018-04-17 12:03:03 -04:00
David Steele
dce11ce3cf Generate code counts for all source files.
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00
David Steele
3e22977484 Use pip 9.03 in test VMs.
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
2018-04-16 15:25:50 -04:00
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier.
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
4744eb9387 Add storagePathRemove() and use it in the Perl Posix driver.
This implementation should be faster because it does not stat each file.  It simply assumes that most directory entries are files so attempts an unlink() first.  If the entry is reported by error codes to be a directory then it attempts an rmdir().
2018-04-11 08:21:09 -04:00
David Steele
c9ce20d41a Skip version checks when testing in --dev mode. 2018-04-09 16:46:36 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module.
These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all.
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 12:16:32 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml.
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 10:19:24 -04:00
David Steele
b77b425096 Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave. 2018-04-07 19:44:00 -04:00
David Steele
82751b3b51 Split debug and assert code into separate headers.
Assert can be used earlier because it only depends on the error-handler and not logging.
2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad More C type cleanup.
* Remove typec.h entirely.
* Order all typdefs above local includes.
2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3 Allow MemContext objects to be copied to a new parent.
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4 Cleanup C types.
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
2018-04-05 22:38:11 -04:00
David Steele
bd25248df0 Abstract Posix storage driver code into a separate module. 2018-04-05 14:48:16 -04:00
David Steele
348278bb68 Make backup directory sync more efficient.
Scanning the entire backup directory can be very expensive if there are a lot of small tables. The backup manifest contains the backup directory list so use it to perform syncs instead of scanning the backup directory.
2018-04-03 21:30:15 -04:00
David Steele
0dce8ba2fa Make backup.history sync more efficient.
Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.
2018-04-03 20:54:18 -04:00
David Steele
93fdb98d15 Storage object improvements.
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
David Steele
90f980fe91 Remove documentation describing info --output=json as experimental. 2018-03-30 21:33:30 -04:00
David Steele
635caff573 Add ASSERT() that is preserved in production builds. 2018-03-30 19:10:34 -04:00
David Steele
a8721ffe11 Improve logic for smart builds to include version changes. 2018-03-27 22:48:22 -04:00
David Steele
5518a2ac51 Full branch coverage for storage/storage module. 2018-03-27 20:40:33 -04:00
David Steele
c1ab7b3c98 Start work on C handle io object and use it to output help. 2018-03-25 11:39:24 -04:00
David Steele
9001b9b957 Replace THROW_ON_SYS_ERROR() with THROW_SYS_ERROR().
The former macro was hiding missing branch coverage for critical error handling.
2018-03-24 14:11:29 -04:00
David Steele
dabf727d91 Include VM type in gcov path to avoid conflicts between VMs with different architectures. 2018-03-21 16:08:08 -04:00
David Steele
c6457b070f Full branch coverage for config/parse module. 2018-03-21 13:46:08 -04:00
David Steele
b234f43c9d Refactor usec to msec in common/time.c.
The implementation provides usec resolution but this is not needed in practice and it makes the interface more complicated due to the extra zeros.
2018-03-21 09:18:48 -04:00
David Steele
31830bdc55 Full branch coverage for all common/type modules. 2018-03-20 16:08:52 -04:00
David Steele
a2030da200 Specify prefix for lcov output.
This prevents pages from moving around depending on the subtest.
2018-03-20 16:01:38 -04:00
David Steele
f4ed36d128 Add switch() to lcov branch exclusions.
lcov requires default to show complete coverage but --Wswitch-enum enforces all enum values be present so no default is needed.

Add documentation for each branch exclusion.
2018-03-20 10:17:53 -04:00
David Steele
d34142dee8 Update lcov settings.
Mark any coverage less that 100% as yellow, less that 90% as red.

Add comments to config file and make genhtml use it as well as lcov.
2018-03-20 09:47:51 -04:00
David Steele
07f38f584a Use lcov for C unit test coverage reporting.
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.

Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
2018-03-19 23:33:28 -04:00
David Steele
45c6a0e864 Remove hard-coded release version from loadTest.c. 2018-03-19 21:03:42 -04:00
David Steele
5b0748cf93 Always autogenerate code if --smart is not specified. 2018-03-19 20:25:55 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present.
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
2018-03-19 19:54:03 -04:00
David Steele
8e9e97728c Remove --smart from --expect tests.
This ensures that new binaries are built before running the tests.
2018-03-19 17:38:44 -04:00
David Steele
2c1687721e Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected.
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.

Reported by Clinton Adams.
2018-03-19 13:08:42 -04:00
David Steele
0c313713b1 Use clang for static code analysis during lint testing.
Nothing found except for some functions that should have been marked __noreturn__.
2018-03-18 13:32:19 -04:00
David Steele
f0451c1494 Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings. 2018-03-15 13:53:03 -04:00
David Steele
8a1ce42c30 Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line.
Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem.

Update command documentation to omit secure options that cannot be specified on the command-line.

Reported by Brad Nicholson.
2018-03-15 12:02:09 -04:00
David Steele
dd3ce70810 Config test code writes secure options to a file instead of passing on the command-line. 2018-03-15 11:09:46 -04:00
David Steele
4fb75c9cc1 Improve conversion of C exceptions to Exception objects.
Colons in the message would prevent all of the message from being loaded into the Exception object.
2018-03-15 11:03:28 -04:00
David Steele
4b6cd92a4c Improve debugging.
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a Add ASSERT_DEBUG() macro for debugging.
Replace all current assert() calls except in tests that can't use the debug code.
2018-03-12 14:31:22 -04:00
Cynthia Shang
dd31ae832d Port most of Config::Config::configLoad() from Perl to C.
Contributed by Cynthia Shang.
2018-03-12 11:25:20 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
94494a632f Update pip before installing awscli. 2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
b987f05f35 Improve bin and libc build performance. 2018-03-09 21:24:30 -05:00
David Steele
c32ee34612 Build with -DNDEBUG by default but disable for testing. 2018-03-09 18:28:34 -05:00
David Steele
fbe4c40386 Move logic for setting Perl configuration defaults to C. 2018-03-09 09:30:50 -05:00
David Steele
919635b6f4 Set config before Main::main() call to avoid secrets being exposed in a stack trace. 2018-03-08 16:24:16 -05:00
Cynthia Shang
34898ce471 Error when multiple arguments are set in the config file for an option that does not accept multiple arguments.
Contributed by Cynthia Shang.
2018-03-08 14:15:05 -05:00
David Steele
e331564c90 Add new test for Common::Io::Process.
Show that output on stderr will raise an exception on close() even if the exit code is 0.
2018-03-08 12:36:55 -05:00
David Steele
62a9bdbd4c Fix issue passing --no-config to embedded Perl.
Reported by Ibrahim Edib Kokdemir.
2018-03-08 10:43:09 -05:00
Cynthia Shang
45ffe37e4f Add cfgOptionTest().
Update cfgOption() calls that are better implemented as cfgOptionTest().

Contributed by Cynthia Shang.
2018-03-05 18:51:48 -05:00
David Steele
c07aa2ae19 Improve code documentation in config module. 2018-03-02 22:42:02 -05:00
David Steele
406acf9cba Enable -Wconversion and silence new warnings. 2018-03-02 12:07:12 -05:00
David Steele
af805af044 Remove Debian package patch now that it has been merged upstream. 2018-03-01 16:59:13 -05:00
David Steele
5d1722b8ef Disable console display of coverage for C files.
Devel::Cover does not handle it well.
2018-03-01 13:32:51 -05:00
David Steele
57d70a5649 Remove RHEL package patch since upstream has been updated. 2018-03-01 13:31:03 -05:00
David Steele
5b491b3035 Add CentOS/RHEL package builds. 2018-02-26 19:32:27 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
Cynthia Shang
f75ba7db94 Update backup host to repository host in the documentation.
Contributed by Cynthia Shang.
2018-02-19 13:17:58 -05:00
Cynthia Shang
4352407777 Option rename cleanup based on review.
* Remove --reset tags from v1 options.
* Use constants for repo prefix.
* Specify CFGDEF_INDEX_PG in option structure rather than adding in code.
* Fix error message references to "backup host".

Contributed by Cynthia Shang.
2018-02-19 10:12:51 -05:00
David Steele
9070325760 Improve performance of HTTPS client.
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
2018-02-18 14:53:27 -05:00
David Steele
27678f6188 Clean up and document Makefile. 2018-02-18 10:44:58 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp().
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
81952c41f4 Report coverage errors via the console.
This helps with debugging coverage issues on remote services like Travis.
2018-02-14 17:13:18 -05:00
David Steele
5f2884cb29 Suppress coverage failures for Archive/Push/Async on Travis.
The coverage report shows some code as never being run -- but that makes no sense because the tests pass.  This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.

Suppress for now with a plan to implement LCOV for the C unit tests.
2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d Build C binary/library for C unit tests that require Perl. 2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48 Move Perl option JSON generation to a new module.
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5 Generate suppressions when running Valgrind.
This makes it easier to suppress errors that are only occurring on Travis.
2018-02-14 09:18:27 -05:00
Cynthia Shang
00e9aca872 Fixed issue where --reset-* was not passing the default value to Perl.
Contributed by Cynthia Shang.
2018-02-13 15:58:14 -05:00
David Steele
7c58fe7451 Use 64-bit integers for config options.
In particular, archive-queue-max requires a greater range than is provided by a 32-bit integer.
2018-02-09 13:54:33 -05:00
David Steele
305a3e1761 Add int64 variant type. 2018-02-09 13:52:02 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml.
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5 Update URL for Debian package repository. 2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
808b6000eb Fix some bugs in C config code:
1) Error when the same option is defined multiple times in a section using alternate names.
2) Fix logging of invalid command error.
3) Warn when a stanza-only option is in a global section.

Also, make a note to add validation of section names to the check command.

Per review by Cynthia Shang.
2018-02-05 15:48:26 -05:00
David Steele
d35757c05f Fix missing log-level-stderr option for stanza-* commands. 2018-02-05 15:19:28 -05:00
David Steele
c3f47bf240 Move config parsing out of Perl tests.
cfgParseTest() is provided in the C library for parsing configs in unit tests.
2018-02-05 12:32:30 -05:00
David Steele
db21b7a360 Cleanup usage of internal options.
Apply internal to options that need to be read to determine locality but should not appear in the help.
2018-02-05 11:50:09 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists.
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
David Steele
298b33988c Fix non-compliant formatting for function declarations.
Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
Cynthia Shang
bacc08792d Ensure latest db-id is selected on when matching archive.info to backup.info.
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).

Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include.
Fixed by Cynthia Shang.
2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl.
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05 Improve speed of C unit tests.
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
2018-01-26 16:41:17 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
404ec30d03 Move Perl help test to real module. 2018-01-23 12:58:35 -05:00
David Steele
3dd882a79e Add internal flag to config definition. 2018-01-23 12:49:04 -05:00
David Steele
41bd5e8148 Improve KeyValue object.
Add kvGetList() to get a value as a list (of 1) even if it is a scalar.
2018-01-23 12:40:02 -05:00
David Steele
2afb73da1c Improve String, StringList, and List objects.
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b The archive-push command is now partially coded in C.
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
David Steele
1f39a34abc Move lock release later in exitSafe().
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
2018-01-17 10:31:21 -05:00
David Steele
7c73f4f6c9 Add command and exit modules. 2018-01-17 09:15:51 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
eb452c8add Improvements to Storage object.
* storagePath() and storageList() commands
* Helper functions for local and spool storag
2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f Improve error module.
Add functions to convert error codes to C errors and handle system errors.
2018-01-16 13:29:27 -05:00
David Steele
bffc6c49b3 Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work. 2018-01-16 13:23:08 -05:00
David Steele
d633417e5d Add Wait object and time module. 2018-01-16 13:05:00 -05:00
David Steele
721d1127a0 Add RegExp object. 2018-01-16 12:57:23 -05:00
David Steele
05c731692b Add varStrForce() to Variant object. 2018-01-16 12:54:06 -05:00
David Steele
83ddd75d82 Improvements to String object.
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
2018-01-16 12:47:08 -05:00
David Steele
d992851456 Rename Perl tests so they don't conflict with their C counterparts. 2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
d8c575801d Add Ini object. 2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff Add Storage object. 2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43 Add KeyValue object. 2017-12-22 22:05:37 -05:00
David Steele
2add6cef95 Add Variant and VariantList objects. 2017-12-22 21:50:27 -05:00
David Steele
c4b30af645 Add Buffer object. 2017-12-22 19:56:01 -05:00
David Steele
f89b027dec Improve String and StringList modules. 2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e Run valgrind on all C unit tests.
Also refactor code to make valgrind happy.
2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c No longer run master branch through CI.
The integration branch will be run through CI and then pushed to master with github status checks.
2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster.
Fixed by Cynthia Shang.
Reported by Stephen Frost.
2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10.
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built.
Contributed by Cynthia Shang.
2017-12-13 11:16:27 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated.
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore.
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
4f1725dc36 Replace cfgCommandTotal()/cfgOptionTotal() functions with constants.
The constants are applicable in more cases and allow the compiler to optimize certain loops more efficiently.
2017-12-05 09:45:51 -05:00
David Steele
59fd92fdd5 Move restore test infrastructure to HostBackup.pm.
Required to test restores on the backup server, a fairly common scenario.

Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-03 17:08:49 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb Add unit tests for the Manifest module.
Also minor changes to Manifest module, mostly for test reproducibility.

Contributed by Cynthia Shang.
2017-11-28 11:44:24 -05:00
David Steele
4a0eb6863d Sync time to prevent build failures when running on VirtualBox. 2017-11-27 22:16:24 -05:00
David Steele
332861c6c9 Only build C binary/library for Perl unit/integration tests. 2017-11-27 21:19:59 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl.
This allows certain time-critical commands (like async archive-push) to run more quickly.
2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18 Add List and StringList objects to simplify list handling. 2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b Add String object to simplify string handling. 2017-11-26 17:49:10 -05:00
David Steele
7cf955425e The C library is now required.
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
c164fc5fb9 Add memGrowRaw() to memory context module. 2017-11-26 12:22:50 -05:00
David Steele
ac54aac448 Fixed an issue passing parameters to remotes.
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
2017-11-21 13:17:44 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function.
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0.
The filter was used with compress level set to 0 which added overhead without any benefit.
2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore.
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 12:46:35 -05:00
David Steele
8ccfaa9ad3 Designate a single distro (Ubuntu 16.04) for C coverage testing
Enable optimizations on the distros that don't do coverage testing.
2017-11-20 09:05:55 -05:00
David Steele
deae0b8cd7 Refactor C page checksum unit test to compile with -Wstrict-aliasing. 2017-11-20 08:58:47 -05:00
David Steele
5d741ea9a9 Refactor C unit test macros so they compile with -Wstrict-aliasing. 2017-11-19 16:46:31 -05:00
David Steele
3b43047e0d Disable -Wclobber compiler warning.
...because it is mostly useless but keep the rest of of -Wextra.
2017-11-19 16:44:33 -05:00
David Steele
dc1a5c18ac Wrap try in a do...while loop to make sure that no random else is attached to the main if block. 2017-11-19 16:30:23 -05:00
David Steele
9395ad7043 Improve base64 implementation.
Different encoded strings could be generated based on compiler optimizations. Even though decoding was still successful the encoded strings did not match the standard.
2017-11-18 22:49:43 -05:00
David Steele
500d6b4b66 Automated tests for 32-bit i386/i686 architecture. 2017-11-18 20:14:14 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files.
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-18 20:02:54 -05:00
David Steele
b701cd5a5f Designate a single distro (Ubuntu 16.04) for Perl coverage testing.
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions.
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test.
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
2017-11-15 13:17:57 -05:00
David Steele
22fe684b1f Only check expect logs on CentOS 7.
Variations in distros cause false negatives in tests but don't add much value.
2017-11-14 17:16:39 -05:00
David Steele
b8746f368d Inflate performance improvement for gzip filter and full unit test coverage. 2017-11-14 15:12:31 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore.
Reported by Craig A. James.
2017-11-14 14:48:44 -05:00
David Steele
cf7da546a3 Simplify try..catch..finally names. 2017-11-13 21:22:13 -05:00
David Steele
bcdfc7d0b5 Enable additional warnings for C builds. 2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a Repository encryption support.
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
55f68ef013 Remove configurable option hints.
db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.
2017-10-30 10:58:04 -04:00
David Steele
1ef27ec8c2 Add list type for options.
The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.
2017-10-30 10:50:35 -04:00
David Steele
f57e376c44 Update Debian/Ubuntu containers to download lastest version of pip. 2017-10-30 09:29:46 -04:00
David Steele
8674a4f7ae Allow functions with sensitive options to be logged at debug level with redactions.
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
2017-10-24 12:35:36 -04:00
David Steele
b2cc65d886 Improve performance of test code.
Wait when all tests have been assigned to reduce CPU load.
2017-10-24 10:35:42 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored.
Reported by Jens Wilke.
2017-10-22 18:05:46 -04:00
David Steele
3553ccae44 Add I/O performance tests. 2017-10-22 13:27:47 -04:00
David Steele
9b98c2e7d6 Run all tests on tempfs rather than local disk. 2017-10-22 13:11:03 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
44c7c0475a Improve protocol error handling.
In particular, "stop" errors are no longer reported as "unexpected".
2017-10-21 14:11:33 -04:00
David Steele
f4524aeaa9 Add base64 encode/decode. 2017-10-18 11:14:26 -04:00
Cynthia Shang
2d56de00fc Fix instances where database-id was not rendered as an integer in JSON info output.
Fixed by Cynthia Shang.
Reported by Jason O'Donnell.
2017-10-18 08:42:32 -04:00
David Steele
b1efd598cf Add PostgreSQL versions to Debian VMs for testing. 2017-10-16 18:21:47 -04:00
David Steele
c94e52f697 Remove error when overlapping timelines are detected.
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.

Reported by blogh.
2017-10-16 14:34:44 -04:00
David Steele
583a76f605 Add C memory contexts. 2017-10-16 11:25:49 -04:00
David Steele
6f5186f9e6 Convert config tests into C unit tests. 2017-10-16 11:13:51 -04:00
David Steele
9d78948a14 Improvements to page checksum module and tests:
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
2017-10-16 11:03:06 -04:00
David Steele
80758f0023 Add test macros for C results and errors. 2017-10-16 10:54:35 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler.
Basic try ... catch ... finally pattern to simplify error handling in C.
2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5 Add C unit test infrastructure. 2017-10-12 12:55:48 -04:00
David Steele
abea4d1fd5 Remove Debian test repo after PostgreSQL 10 release. 2017-10-05 12:33:13 -04:00
David Steele
d89b7d5862 Warnings in C builds treated as errors. 2017-09-30 10:48:25 -04:00
David Steele
40a85e34f1 Improve C library smart build.
Ignore changes outside of /lib/pgBackRest/Config.
2017-09-20 19:57:05 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
7c95a2f4de Updates to installation documentation.
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
2017-09-08 21:50:00 -07:00
David Steele
6343fdd584 Additional backup exclusions.
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 08:26:57 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways.
Suggested by Victor Gdalevich.
2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.`
Contributed by Cynthia Shang.
2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files.
Reported by Jesper St John, Aleksandr Rogozin.
2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
f8de73674c Updated Docker build in Vagrantfile. 2017-08-10 20:52:29 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
2017-08-08 18:09:07 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test.
Contributed by Cynthia Shang.
2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade.
Fixed by Cynthia Shang.
2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
fa992433da Update CI code to prevent deprecated VM from being rewritten. 2017-07-27 11:31:42 -04:00
David Steele
14d6974d1b Use Google DNS in test environment for consistency. 2017-07-26 13:40:09 -04:00
David Steele
df7d633683 Improvements to command/command-line help:
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
David Steele
cbb749e12a Debian/Ubuntu documentation now builds on Ubuntu 16. 2017-07-24 21:12:05 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups.
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements:
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
2017-06-24 10:59:00 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
6327799c9a Use old trusty image until new image works with RHEL6. 2017-06-22 18:13:59 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00
David Steele
f8849d7359 Remove patch directory before Debian package builds.
Patches should be merged into master before trying a new build after a release with patches.
2017-06-22 12:37:21 -04:00
David Steele
8b9965abcb Remove process-max option.
Parallelism is now tested in a more targeted manner and the high level option is no longer needed.
2017-06-21 19:21:16 -04:00
David Steele
2348693389 Reduce process redundancy in mock tests. 2017-06-21 18:47:25 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
73274fc608 Remove Ubuntu 12.04 from CI now that it is EOL (it can still be run manually). 2017-06-21 17:02:01 -04:00
David Steele
c69baa2de1 Install sudo in base containers rather than on demand. 2017-06-21 16:51:41 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
edada21866 Rename archive-push unit test.
This is part of the archive consolidation but is committed separately so that git will follow the rename.
2017-06-21 08:45:58 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor.
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
2017-06-15 15:32:10 -04:00
David Steele
269ae4556d Fix timeouts in ExecuteTest to speed multi-process testing. 2017-06-15 15:20:02 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process.
Fixed by Cynthia Shang.
2017-05-16 11:28:51 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable.
Fixed by Cynthia Shang.
2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version.
Fixed by Cynthia Shang.
2017-05-12 15:49:14 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
Suggested by Yogesh Sharma.
2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid.
Suggested by Yogesh Sharma.
2017-04-12 09:53:07 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files.
In both cases the files are checksummed on resume so there's no danger of partial copies.
2017-04-12 09:05:57 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
David Steele
8546571461 Ignore clock skew in container libc/package builds using make.
It is common for containers to have clock skew so the build process takes care of this issue independently.
2017-04-10 17:20:21 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages.
Contributed by Cynthia Shang.
2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers.
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring.
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
2017-04-07 10:31:49 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade.
Contributed by Cynthia Shang.
2017-04-06 19:36:37 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash.
Refactor File->list() and fileList() to accept optional parameters.
2017-04-04 21:17:19 -04:00
David Steele
9eae36a702 Fixed flapping archive stop tests. 2017-04-04 21:15:22 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
Contributed by Cynthia Shang.
2017-03-22 19:03:28 -04:00
David Steele
cbd879a983 Updated vagrant to new version and image. 2017-03-14 22:05:17 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings.
Reported by Stephen Frost.
2017-03-13 15:50:13 -04:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned.
Reported by blogh.
2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements:
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
2017-03-02 11:47:23 -05:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment.
Reported by Stephen Frost.
2017-03-01 14:15:46 -05:00
David Steele
774b243719 Fixed dev version string in Debian changelog. 2017-03-01 12:22:47 -05:00
David Steele
2c48cf8d0d Fixed Vagrantfile to match user/group changes made for Travis CI. 2017-03-01 09:36:33 -05:00
David Steele
8161ab1c50 Temporarily disable flapping keep-alive test. 2017-02-26 12:58:48 -05:00
David Steele
abe0935ea5 Disable CI config writing with —no-ci-config. 2017-02-26 12:53:41 -05:00
David Steele
ac485cc355 Change —retry default to 0. 2017-02-22 18:43:10 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
023578c008 Make package tests work with dev builds. 2017-02-21 14:22:03 -05:00
David Steele
0e9bc513c9 Suppress lint on all but the first test. 2017-02-21 12:54:27 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest().
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-02-21 11:56:55 -05:00
David Steele
01312f6260 Improved formatting of testResult() output. 2017-02-21 11:47:45 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
56543f7a99 Added package builds to test suite and other improvements:
* Automated builds of Debian packages for all supported distributions.
* Added --dev option to aggregate commonly used dev options.
* Added --no-package option to skip package builds.
* C library and packages are built by default, added -smart option to rebuild only when file changes are detected.
* The --libc-only option has been changed to --build-only now that packages builds have been added.
2017-02-17 22:31:16 -05:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
a54018d573 Improved performance of wal archive min/max provided by the info command.
Suggested by Jens Wilke.
2017-02-13 10:03:02 -05:00
David Steele
bc2bb040c5 Fixed an issue where an archive-push error would not be retried.
It would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted).

Reported by Jens Wilke.
2017-02-13 09:57:10 -05:00
David Steele
498f52da09 Improvements to documentation engine:
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
2017-02-10 10:22:05 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
1666ee177f Fixed --no-online tests to suppress expected errors. 2017-02-02 20:44:42 -05:00
David Steele
7418559a67 Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination. 2017-01-31 10:36:59 -05:00
David Steele
eb1a1e5308 Missed expect logs from commit 455c604. 2017-01-30 16:22:12 -05:00
David Steele
34e4cfaeec Fixed minor typos. 2017-01-30 15:48:10 -05:00
David Steele
455c6048ee C Library builds immune to clock differences between VM and host. 2017-01-30 14:00:40 -05:00
David Steele
6b99d770b5 Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN. 2017-01-30 13:59:00 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
18db178ac3 Various improvements to the test suite:
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
2017-01-27 09:42:30 -05:00
David Steele
7d1347cdec Split the archive-get and archive-push commands into separate modules. 2017-01-10 19:59:32 -05:00
David Steele
b58045f575 Moved the Archive modules to the Archive directory. 2017-01-10 19:51:20 -05:00
David Steele
2ab63f27b2 C Library builds only run when C library has actually changed. 2017-01-10 13:12:53 -05:00
David Steele
0b4055e588 Moved File->manifest() into the FileCommon.pm module. 2017-01-10 13:08:03 -05:00
David Steele
0e4f51c271 Added integration for testing coverage with Devel::Cover. 2017-01-09 20:49:04 -05:00
David Steele
39744a4f1d Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner().
Reported by Leonardo Avellar.
2017-01-05 15:19:56 -05:00
David Steele
4ff2714bf0 Return proper error code when unable to convert a relative path to an absolute path.
Suggested by Yogesh Sharma.
2017-01-04 18:39:50 -05:00
David Steele
c8dfc673b8 Moved tests into new modules to better reflect actual usage. 2017-01-04 18:31:16 -05:00
David Steele
e3d63c0252 Allow multiple --module, --test, and --run options to be used for test.pl. 2017-01-04 18:29:13 -05:00
Cynthia Shang
be337c951a Improvements to stanza-create:
* Force only required when a change is needed for backup.info or archive.info.
* Identical files are not overwritten.
2017-01-04 10:12:29 -05:00
David Steele
b91ca11329 Added expect log expression to replace year subdirectories in backup.history. 2017-01-04 10:09:21 -05:00
David Steele
83beab7ec3 Refactor File module to improve test coverage.
Added unit tests for low-level functions in the BackupCommon module.
2016-12-23 11:43:26 -05:00
David Steele
5d3473b52d Refactor File module to improve test coverage. 2016-12-23 08:30:34 -05:00
David Steele
6b2666a9d7 Split test modules into separate files.
Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 08:22:59 -05:00
Cynthia Shang
56144c99c0 Improved check command to run on a standby.
Only basic checks are done because pg_switch_xlog() cannot be executed on a replica.

Contributed by Cynthia Shang.
2016-12-21 14:35:20 -05:00
Cynthia Shang
974a02dbe4 Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory.
Contributed by Cynthia Shang.
2016-12-20 16:52:20 -05:00
David Steele
36a5349b1c Added the --checksum-page option.
This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-12 18:54:07 -05:00
David Steele
6dd0829033 File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process. 2016-12-12 18:47:17 -05:00
David Steele
b4884e5a0b Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database.
Reported by Michael Vitale.
2016-12-10 14:05:28 -05:00
David Steele
e61ef6ebce Add more realistic data files to synthetic backup and restore tests. 2016-12-10 09:13:55 -05:00
David Steele
eff7b46eb1 Various improvements to validation of backup and restore. 2016-12-10 09:11:12 -05:00
David Steele
f818901c0e Added the --repo-sync option to allow directory syncs in the repository to be disabled.
This is for file systems that do not support them, e.g. NTFS
2016-12-10 09:06:45 -05:00
David Steele
896984848b Improved errors returned from child processes by removing redundant error level and code. 2016-12-04 17:51:00 -05:00
David Steele
817592837c Added a predictable log entry to signal that a command has completed successfully.
For example a backup ends successfully with: INFO: backup command end: completed successfully.
2016-12-04 14:44:53 -05:00
David Steele
f823a25ac2 Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error.
* The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error.
* Removed erroneous --no-config option in help test module.
* Changed the --no-fork test option to --fork with negation to match all other boolean parameters.
2016-12-04 09:25:40 -05:00
David Steele
d18cd7ad0f Fixed an issue that prevented errors from being output to the console before the logging system was initialized.
That is, while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not.
2016-12-03 18:01:17 -05:00
David Steele
2e01e6b9e1 Improved consistency and flexibility of the protocol layer by using JSON for all messages. 2016-12-03 17:34:51 -05:00
Cynthia Shang
b7f1738200 Code cleanup in preparation for improved stanza-create command.
Contributed by Cynthia Shang.
2016-12-03 10:28:08 -05:00
David Steele
005fc08795 Added the --repo-link option.
Allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
2016-12-03 10:23:26 -05:00
David Steele
6ae4d7de81 Fixed a bug where internal symlinks were not being created for tablespaces in the repository.
This issue was only apparent when trying to bring up clusters in-place with filesystem snapshots and did not affect normal backup and restore.
2016-12-03 09:56:05 -05:00
Cynthia Shang
22cd29771f Update control and wal test files to 9.4 with matching system identifiers.
Contributed by Cynthia Shang.
2016-12-01 09:33:14 -05:00
David Steele
a850335015 Simplified the result hash of File->manifest(), Db->tablespaceMapGet(), and Db->databaseMapGet(). 2016-11-30 14:36:39 -05:00
David Steele
09df07efb7 For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process.
The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
2016-11-30 14:23:05 -05:00
Cynthia Shang
e951c4f9c2 Abstracted code to determine which database cluster is the master and which are standbys.
Contributed by Cynthia Shang.
2016-11-30 14:15:11 -05:00
David Steele
d101e9c84e Improved exception handling in file unit tests. 2016-11-30 14:03:58 -05:00
David Steele
c9b49b0d7e Doc engine improvements.
Bug Fixes:

* Fixed and issue that suppressed exceptions in PDF builds.

Features:

* Allow a source to be included as a section so large documents can be broken up.
* Added section link support to Markdown output.
* Added list support to PDF output.
* Added include option to explicitly build sources (complements the exclude option though both cannot be used in the same invocation).
* Added keyword-add option to add keywords without overriding the default keyword.
* Added debug option to doc.pl to easily add the debug keyword to documentation builds.
* Added pre option to doc.pl to easily add the pre keyword to documentation builds.

Refactoring:

* Improvements to markdown rendering.
* Remove code dependency on project variable, instead use title param.
2016-11-17 16:35:11 -05:00
David Steele
dbb9d80dab Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches.
This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode.

Reported by Stephen Frost.
2016-11-17 09:39:21 -05:00
David Steele
9d9c438f59 Doc engine improvements.
Bug Fixes:

* Fixed missing variable replacements.
* Removed hard-coded host names from configuration file paths.

Documentation Features:

* Allow command-line length to be configured using cmd-line-len param.
* Added compact param to allow CSS to be embedded in HTML file.
* Added pretty param to produce HTML with proper indenting.
* Only generate HTML menu when required and don't require index page.
* Assign numbers to sections by default.
* VM mount points are now optional.
2016-11-07 17:06:35 +02:00
David Steele
dd621081b9 Fixed an issue where tablespace paths with the same prefix would cause an invalid link error.
Reported by Nikhilchandra Kulkarni.
2016-11-07 16:37:16 +02:00
David Steele
21ca412702 Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed. 2016-11-05 19:22:30 +02:00
David Steele
f43e5bc52d Removed extraneous use lib directives from Perl modules.
Suggested by Devrim Gündüz.
2016-11-04 13:56:26 +02:00
David Steele
171c5835bb Another expect log update for stanza-create feature. 2016-11-03 17:48:59 +02:00
David Steele
26f76762b5 Expect log updates for stanza-create feature. 2016-10-17 10:03:42 +01:00
Cynthia Shang
8d1880ca4a Added the stanza-create command to formalize creation of stanzas in the repository.
Contributed by Cynthia Shang.
2016-10-14 12:21:47 +01:00
David Steele
16d0e767ef Set application_name to "pgBackRest [command]" for database connections.
Suggested by Jens Wilke.
2016-10-05 21:15:10 -04:00
David Steele
0a4099fff8 Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock.
Suggested by Jens Wilke.
2016-10-05 21:13:29 -04:00
David Steele
5dc801d59d Fixed an issue where the async archiver would not be started if archive-push did not have enough space to queue a new WAL segment.
This meant that the queue would never be cleared without manual intervention (such as calling archive-push directly). PostgreSQL now receives errors when there is not enough space to store new WAL segments but the async process will still be started so that space is eventually freed.

Reported by Jens Wilke.
2016-10-05 09:47:34 -04:00
David Steele
2a18674111 Added the log-level-stderr option.
Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior.

Suggested by Sascha Biberhofer.
2016-10-05 09:09:30 -04:00
David Steele
d81198e65e Added the cmd-ssh to allow the ssh client to be specified.
Suggested by Jens Wilke.
2016-10-05 08:55:54 -04:00
David Steele
6f34fbc695 Vagrant build improvements:
* Miminize TeXLive package list to save time during VM builds.
* General cleanup and added timestamps.
* Pin to specific box version.
2016-10-03 09:15:29 -04:00
David Steele
1491862716 Regression tests can now be run as any properly-configured user, not just vagrant. 2016-10-01 13:39:44 -04:00
David Steele
156859c5d5 Update CentOS/Debian package definitions. 2016-09-29 19:07:46 -04:00
David Steele
3e4ca153ce Fixed broken vm-max option in test.pl. 2016-09-29 18:26:32 -04:00
David Steele
5d033d028c Archive validation improvements:
* Fixed error message to properly display the archive command when an invalid archive command is detected.
* Check that archive_mode is enabled when archive-check option enabled.
2016-09-28 19:45:33 -04:00
David Steele
25f40f9b15 Log improvements:
* pgBackRest version number included in command start INFO log output.
* Process ID logged for local process start/stop INFO log output.
* Fixed missing expect output for help module.
2016-09-17 11:53:28 -04:00
David Steele
f54145c0cc Fixed timeout issues.
* Fixed an issue where local processes were not disconnecting when complete and could later timeout. (Reported by Todd Vernick.)
* Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive. (Reported by Todd Vernick.)
2016-09-14 16:37:07 -05:00
David Steele
e08b56a4e3 Show Process ID in log output instead of filtering it out with the timestamp. 2016-09-14 16:30:57 -05:00
John K. Harvey
f253a4d85c Suppress "dpkg-reconfigure: unable to re-open stdin: No file or directory" warning in Vagrant VM build.
Contributed by John Harvey.
2016-09-08 17:29:26 -04:00
Cynthia Shang
d2add293c7 Archive retention improvements:
* Fixed an issue where retention-archive was not automatically being set when retention-archive-type=diff, resulting in a less aggressive than intended expiration of archive.
* Additional warnings when archive retention settings may not have the intended effect or would allow indefinite retention.
* Closed #235: "Retention policy question" by adding documentation for archive retention.

Contributed by Cynthia Shang.
2016-09-07 08:07:37 -04:00
David Steele
cc9c8819ee Make uid/gid selection for containers dynamic. 2016-09-06 14:01:39 -04:00
David Steele
cedfcbd4be Update Vagrantfile:
* Change box provider to bento (hoping for better stability with Docker).
* Install texlive from Debian packages.
2016-09-06 11:02:01 -04:00
David Steele
dd8d781217 Safely check eval blocks and convert $@ to $EVAL_ERROR. 2016-09-06 09:44:50 -04:00
David Steele
a701309453 Converted Perl threads to processes. 2016-09-06 09:35:02 -04:00
David Steele
4026ad649d Refactor of protocol minions in preparation for the new local minion. 2016-09-06 09:15:22 -04:00
David Steele
f71073c11f Improved temp file handling for archiving. 2016-09-04 19:29:30 -04:00
David Steele
8f61473955 Improve ASSERT error handling. 2016-09-04 09:25:20 -04:00
David Steele
f986e6ae33 File module improvements.
* Remove obsolete thread index variable from File() module.
* Changed temporary file names to consistently use .pgbackrest.tmp.
2016-09-04 09:22:53 -04:00
David Steele
bcdb5cdac8 Fixed a issue where tablespaces were copied from the master during standby backup. 2016-09-04 09:19:44 -04:00
David Steele
2feaaf225e Exclude contents of $PGDATA/pg_replslot directory. 2016-09-04 09:13:13 -04:00
Cynthia Shang
634f5c6829 Closed #250: Fixed the check command so backup info is checked remotely and not just locally.
Fixed by Cynthia Shang.
2016-08-29 14:09:37 -04:00
David Steele
c9bdf6a9ae Fixed timeout issue.
The protocol-timeout option was not automatically increased when the db-timeout option was increased.

Reported by Todd Vernick.
2016-08-25 11:27:00 -04:00
David Steele
5ada189a92 Backup from a standby cluster.
A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 11:25:46 -04:00
David Steele
d0b6f78b20 More flexible configuration for databases
Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 12:39:27 -04:00
David Steele
cd6278e5af Revert some backup exclusions until they have been tested more thoroughly. 2016-08-24 12:27:48 -04:00
David Steele
9c13ca1d09 mprove backup log messages to indicated which host the files are being copied from. 2016-08-24 12:01:08 -04:00
David Steele
f1412baccf Exclude directories during backup that are cleaned, recreated, or zeroed by PostgreSQL at startup.
These include (depending on the version where they were introduced): pgsql_tmp, pg_dynshmem, pg_notify, pg_replslot, pg_serial, pg_snapshots, pg_stat_tmp, pg_subtrans. The postgresql.auto.conf.tmp file is now excluded in addition to files that were already excluded: backup_label.old, postmaster.opts, postmaster.pid, recovery.conf, recovery.done.
2016-08-16 09:35:16 -04:00
David Steele
b25f10949a Fixed #229: Check fails on target db host.
Options that should not allow multiple values could be specified multiple times in pgbackrest.conf without an error being raised.
2016-08-15 20:15:17 -04:00
David Steele
1e0f15f425 Improve error message for links that reference links in manifest build. 2016-08-15 17:23:37 -04:00
David Steele
f9fa1270b2 Fixed #236: Recursive user tablespace symlink.
A tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely.
2016-08-15 17:11:45 -04:00
David Steele
17b79d6279 Database version refactoring.
* Refactor db version constants into a separate module.
* Update synthetic backup tests to PostgreSQL 9.4.
2016-08-11 22:35:24 -04:00
David Steele
bff262ac47 Removed all OP_* function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__. 2016-08-11 17:32:28 -04:00
David Steele
34afe5e85b Fixed issue with tablespace link checking.
* Tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not.
* Also hardened relative path checking a bit.
2016-08-09 09:05:27 -04:00
David Steele
f2b3dda647 Fixed an issue where a remote could try to start its own remote.
This is a better approach than 93320b8 (reverted in this commit) because it ensures that the remote type will be none so any functions that utilize optionRemoteTypeTest will work correctly.

This bug was only an issue when backup-host was not properly configured on the database host.
2016-07-30 08:49:26 -04:00
David Steele
94a4cb4123 Disable control master for older OS versions where it is less stable. 2016-07-30 08:43:38 -04:00
David Steele
1b6b26956a Reduce the frequency that certain tests are run to save time in regression. 2016-07-29 18:46:19 -04:00
David Steele
a3b8808f94 Fixed an issue where the contents of pg_xlog were being copied if the directory was symlinked. 2016-07-29 18:44:53 -04:00
David Steele
ef33febfb7 Temp files created during backup are now placed in the same directory as the target file. 2016-07-29 18:43:59 -04:00
David Steele
95bbc8ad54 Fixed an issue where multi-threaded tests were not being run when requested. 2016-07-29 18:02:16 -04:00
Cynthia Shang
6e9d5f5bd3 Fixed check command to throw an error when database version/id does not match that of the archive.
Fixed by Cynthia Shang.
2016-07-29 14:02:11 -04:00
David Steele
273d1c08d8 Improved remote error messages to identify the host where the error was raised. 2016-07-26 16:57:38 -04:00
David Steele
b6ac6b0f1c Suppress banners on SSH protocol connections. 2016-07-26 16:28:43 -04:00
David Steele
9ae6c01432 Fixed a version checking issue in test.pl. 2016-07-19 15:22:20 -04:00
David Steele
5df2eed729 Order database versions during test.pl --vm-build. 2016-07-02 09:07:48 -04:00
David Steele
418039027b Closed #213: Archive command should not be validate when archive-check=n 2016-07-02 09:02:55 -04:00
David Steele
0f2eb424bb Fix for container build in Vagrantfile. 2016-07-02 08:59:11 -04:00
David Steele
9516db4ad6 Closed #209: Dont' use "localtime" for HTML-docs.
The release-date-static var has been added to allow the html footer to use the last release date instead of localtime().
2016-06-27 20:37:25 -04:00
David Steele
cc2a8777d5 User/group permissions improvements.
Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
2016-06-26 21:01:20 -04:00
David Steele
b571218930 Minor refactor of version variable to a constant.
It had originally been designed to play nice with a specific packaging tool but that tool was never used.
2016-06-26 20:53:45 -04:00
David Steele
bd25223fd6 Rename test paths for clarity.
This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved.  Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 10:54:31 -04:00
David Steele
c8f806a293 Major refactor of the test suite.
* Make the code more modular and object-oriented.
* Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 08:12:58 -04:00
David Steele
23a3911830 Stop using pg_xlogfile_name().
The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
2016-06-24 08:06:20 -04:00
David Steele
41b27c0456 Protocol timeout option and keep-alive fixes.
* Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded operation and were completely absent during single-threaded operation when resuming from a previous incomplete backup.

Reported by Janice Parkinson.

* Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
* Failure to shutdown remotes at the end of the backup no longer throws an exception. A warning is still generated that recommends a higher protocol-timeout.
2016-06-22 18:01:18 -04:00
David Steele
012405a33b Closed #207: Expire fails with unhandled exception.
* Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set.  This was not an issue when expire was run after a backup, which is the usual case.
* Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.

Reported by Chris Barber.
2016-06-18 09:55:00 -04:00
David Steele
cb32a8ab1f VM build improvements.
* Properly set user env before calling test.pl --vm-build.
* Remove squid proxy made obsolete by more efficient builds.
2016-06-13 11:18:06 -04:00
Cynthia Shang
7e45ed8366 Added check command.
The check command validates that pgBackRest is configured correctly for archiving and backups.

Contributed by Cynthia Shang.
2016-06-12 09:13:46 -04:00
David Steele
f1e92804ce Docker container improvements.
* Containers now use a squid proxy for apt/yum to speed builds.
* Obsolete containers are removed by the <br-option>--vm-force</br-option> option.
* Greatly reduced the quantity of Docker containers built by default. Containers are only built for PostgreSQL versions specified in db-minimal and those required to build documentation. Additional containers can be built with --db-version=all or by specifying a version, e.g. --db-version=9.4.
2016-06-12 09:00:16 -04:00
David Steele
3d761b726a Update TeX Live to 2016 version. 2016-06-10 21:22:36 -04:00
David Steele
eaa981c5aa Added execution cache for document generation.
Added an execution cache so that documentation can be generated without setting up the full container environment. This is useful for packaging, keeps the documentation consistent for a release, and speeds up generation when no changes are made in the execution list.
2016-06-02 09:32:56 -04:00
David Steele
174b16165b Replaced overzealous perl -cW check.
It failed on Perl 5.22 so replaced with with perl -cw.
2016-05-26 16:57:35 -04:00
David Steele
28f1c927e7 Upgrade doc/test VM to Ubuntu 16.04.
* This will help catch Perl errors in the doc code since it is not run across multiple OSs like the core and test code.
* It is to be hoped that a newer kernel will make Docker more stable.
2016-05-26 14:03:32 -04:00
David Steele
bfec2d2f89 Fixed release version test when the current version is development. 2016-05-26 11:03:10 -04:00
David Steele
c8d68bcf2d More detailed release notes.
Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated.  An "Additional Notes" section has been added for changes to documentation and the test suite that do not affect the core code.
2016-05-26 10:34:10 -04:00
Cynthia Shang
5a85122841 Moved change log to website.
The change log was the last piece of documentation to be rendered in Markdown only.  Wrote a converter so the document can be output by the standard renderers.  The change log will now be located on the website and has been renamed to "Releases".
2016-05-26 09:20:55 -04:00
David Steele
e7d4a25d6a Added Ubuntu 16.04 and Debian 8 to regression tests.
Also added some new fields to make container generation simpler for distros that share a common package base.
2016-05-24 08:17:13 -04:00
David Steele
f2a14a06e5 Moved version checking code so it doesn't run in containers.
The required modules were not loaded so errors were occurring during regression tests.
2016-05-23 20:04:36 -04:00
David Steele
c604b6e657 Test release version using change-log.xml instead of CHANGELOG.md.
This is in preparation for moving the change log to the website and removing it from Github.
2016-05-23 19:33:02 -04:00
David Steele
0451d3afdd Support for non-exclusive backups in PostgreSQL 9.6. 2016-05-16 17:59:26 -04:00
Cynthia Shang
48eecc2c04 Added vim to test virtual machine. 2016-05-16 16:49:54 -04:00
David Steele
77b01e980f Make the initial capital for hints consistent.
Some hints started with a capital letter and others did not. Make them all lower case for consistency.
2016-05-14 10:36:35 -04:00
David Steele
512d006346 Refactor database version identification for archive and backup commands.
Added database version constants and changed version identification code to use hash tables instead of if-else.  Propagated the db version constants to the rest of the code and in passing fixed some path/filename constants.

Added new regression tests to check that specific files are never copied.
2016-05-14 10:33:12 -04:00
David Steele
0c320e7df7 Allow selective restore of databases from a cluster backup.
This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 09:21:39 -04:00
David Steele
e430f0a054 Added --db-version=minimal option as default.
This change assigns each version of PostgreSQL to a specific OS version for testing to minimize the number of tests being run.  In general, older versions of PostgreSQL are assigned to older OS versions.

The old behavior can be enabled with `--db-version=all`.
2016-05-11 08:59:34 -04:00
David Steele
a6a19e3735 Test directories are now located on the host VM rather than in the Docker container.
This change allows for easier testing since all files are local on the host VM and can be easily accessed without using `docker exec`.  In addition, this change is required to allow multiple Docker containers per test case which is coming soon.
2016-05-10 18:12:37 -04:00
Cynthia Shang
60b901948a Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. 2016-05-06 10:25:46 -04:00
David Steele
7b64843221 Added trim() to process id read from lock file.
The prior way worked fine but added extraneous linefeeds to the log output.
2016-05-06 09:37:01 -04:00
David Steele
dee3e86ff8 README.md is now rendered by new Markdown code. 2016-04-16 11:55:23 -04:00
David Steele
9457e15347 New manifest format.
* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 22:50:02 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00
David Steele
885797e4b5 Migrated many functions from File.pm to FileCommon.pm.
This makes make the migrated file functions available to parts of the code that don't have access to a File object.  They still exist as wrappers in the File object to support remote calls.
2016-04-13 19:09:35 -04:00
David Steele
723abadb75 Change spacing in expect logs to accommodate DETAIL level. 2016-04-13 18:55:34 -04:00
David Steele
64b2858bb5 Test VM build improvements.
* Specific VMs can now be built by using --vm along with --vm-build.
* Docker caching can be disabled with --vm-force.
* ControlMaster is now used for al VMs to improve test speed.
2016-04-13 18:47:29 -04:00
David Steele
8adbcccd02 Closed #184: Change test/log directory to test/expect.
This should make it clearer that these are expect logs for regression testing.
2016-04-06 10:00:11 -04:00
David Steele
1ce9961f8a Closed #183: Options --repo-path and --repo-remote-path ignored in archive-push.
Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.
2016-04-06 08:57:46 -04:00
David Steele
d489914173 Closed #180: Backup format regression in v0.90. 2016-03-22 09:57:39 -04:00
David Steele
d4c46acf48 Working on #174: Get Perl Critic to pass on stern
Added single exception for the violation of BuiltinFunctions::ProhibitStringyEval and removed from policy file.
2016-02-23 09:39:37 -05:00
David Steele
048571e23f Closed #173: Add static source code analysis
Perl Critic added and passes on gentle.  A policy file has been created with some permanent exceptions and a list of policies to be fixed in approximately the order they should be fixed in.
2016-02-23 09:25:22 -05:00
David Steele
d35ab82a83 Closed #167: Rename --no-start-stop option to --no-online. 2016-02-19 19:32:02 -05:00
David Steele
fb63a981cb Closed #166: Copy global/pg_control last during backup. 2016-02-14 20:42:11 -05:00
David Steele
17cbab524a Combine chmod commands in tests for efficiency. 2016-02-11 21:40:43 -05:00
David Steele
de9b6d5133 Closed #113: Better error message when repo does not exist. 2016-02-06 16:49:12 -05:00
David Steele
f8b47997a3 Closed #127: More sanity checking for --delta restores
Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-05 23:03:29 -05:00
David Steele
5c815e4fc0 Closed #58: Get catalog number for better tablespace copying
When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 14:33:15 -05:00
David Steele
02405e851c Closed #163: Document Generation Issue
Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs.

Reported by John Harvey.
2016-01-22 18:01:21 -05:00
David Steele
83b1a3cb44 Closed #150: Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail. 2016-01-20 21:28:40 -05:00
David Steele
747346db67 Closed #162: When restore --set=latest (the default) the actual backup restored will be output to the log. 2016-01-20 18:34:42 -05:00
David Steele
2d38876bc4 Closed #93: The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will still not be possible for backups that fall outside of archive retention. 2016-01-20 17:24:44 -05:00
David Steele
904a8c956f Refactor of expiration to use backup.info for a list of what needs to be expired rather than looking at the backup directory. 2016-01-19 07:39:29 -05:00
David Steele
65cce69dcc Closed #131: Offline backups make expire command fail. 2016-01-15 23:37:17 -05:00
David Steele
fce7261536 Only allow --no-cleanup option when a single test is selected. 2016-01-15 23:22:30 -05:00
David Steele
4d12cc233e Fix for typo in error message. 2016-01-15 20:31:29 -05:00
David Steele
e91e481f3d Closed #19: The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors. 2016-01-15 07:07:00 -05:00
David Steele
67fb7a23bc Master test log is output to file. 2016-01-14 19:37:06 -05:00
David Steele
c2b1620c53 Logging enhancements:
1) Show log output realtime when --max-process=1 or only 1 test is being run.
2) Dry run now shows test output when --vm-out is specified.
2016-01-13 22:43:26 -05:00
David Steele
a0a3d1e97e Closed #122: 9.5 Integration
The archive_mode = always setting is not included but is covered by issue #125.
2016-01-13 20:35:12 -05:00
Jason O'Donnell
c7502f341b Generate an error when archive-check=y but archive_command does not execute pg_backrest. 2016-01-11 09:50:45 -05:00
David Steele
81a2fde21f Fixed typos in test documentation. 2016-01-10 21:07:07 -05:00
David Steele
0f88943b7b Database versions are now run as separate tests. 2016-01-10 19:30:51 -05:00
David Steele
cd9fa6f028 Updated test logs for partial WAL segment support. 2016-01-10 09:19:21 -05:00
David Steele
29102b2887 Added error when archive_mode=always is configured. 2016-01-09 23:08:54 -05:00
David Steele
7f0f8823df Added support for partial WAL segments. 2016-01-09 19:37:28 -05:00
David Steele
e72165a689 Provisional support for PostgreSQL 9.5. No support for archive_mode = 'always' or .partial WAL on a standby. 2016-01-09 10:11:03 -05:00
David Steele
c8f863fbab Text execution improvements:
1) Tests for all operating systems can now be run with a single command.
2) Tests can be run in parallel with --process-max.
3) Container generation now integrated into test.pl
4) Some basic test documentation.
2016-01-09 08:21:53 -05:00
David Steele
4a384ffbdb Added archive-check config test.
Fixed capitalization error in optionboolSetTest().
2016-01-03 20:24:14 -05:00
David Steele
6f6c796fbf Fixed an issue where specifying --no-archive-check would throw a configuration error. 2016-01-03 20:21:30 -05:00
David Steele
7f630aec49 Switched to using docker for all unit tests as this allows for greater automation. 2015-12-29 13:57:10 -05:00
David Steele
ba098d7b91 Fixed an issue where longer-running backups/restores would timeout when remote and threaded.
Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work.  The error messages for timeouts was also improved to make debugging easier.
2015-12-24 10:32:25 -05:00
David Steele
b0a6954671 A number of doc improvements:
1) Started on a general markdown renderer
2) Internal links now work in PDF
3) Improvements to PDF styling
4) Some comment and formatting fixes
5) User guide edits.
2015-12-23 11:04:26 -05:00
David Steele
d5cef05a68 Added rc to version check. 2015-12-21 19:25:50 -05:00
David Steele
a840808bc6 Regression test for issue #159: duplicated log section entries in config file
Add a perl -cW test to prevent similar errors in the future.
2015-12-21 19:25:17 -05:00
David Steele
fa05715dec Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication. 2015-11-22 16:44:01 -05:00
David Steele
8ddfdcdd3b Various fixes and features implemented during doc development
* Better messaging for expiration.
* Fixed already stopped message.
* retention-archive and retention-archive-type now use retention-full and 'full' when not specified.
* Fixed issue where backup-user was required (should default to backrest).
* ExecuteTest now supports retries.
* Fixed issue where log test was not comparing test logs.
* Fixed issue where test logs would not match for ssh connection errors
2015-11-22 14:02:14 -05:00
David Steele
57a06ba3b1 Implemented issue #148: Website with basic user guide. 2015-10-28 10:10:36 +01:00
David Steele
6e170d27e4 Replaced expired CentOS VMs with new ones. 2015-10-08 12:23:37 -04:00
David Steele
ddd4c14773 Implemented issue #129: Stop/Start Commands.
Implemented issue #136: Add stderr back into protocol layer.

There were also miscellaneous bug fixes and improvements to buffering in IO.pm.
2015-10-08 11:43:56 -04:00
David Steele
e488f7bb19 Fixed issue #144: Error on resume when temp files (file.tmp.*) present. 2015-09-22 21:07:49 -04:00
David Steele
e0dafbff7f Improvements to issue #132: Improved command-line help.
Regression tests are now more comprehensive by default.
Better handling for errors in safeExit().
Release notes.
2015-09-09 15:40:54 -04:00
David Steele
d7e3be1ebf Fixed issue #138: Fix --no-start-stop working on running db without --force.
Unable to reproduce this anymore.  It seems to have been fixed with the last round of config changes.  Add regression tests to make sure it doesn't happen again.
2015-09-08 18:29:13 -04:00
David Steele
b17bf31fb6 Implement issue #130: Info files should be updated with new version when pgBackRest is upgraded. 2015-09-08 16:58:00 -04:00
David Steele
59b2261a8e Fix issue #137: Info command throws an exception if no stanzas exist. 2015-09-08 16:44:47 -04:00
David Steele
8d01138f6d Improved issue #135: Suppress PostgreSQL notices.
Regression tests now fail if there is any unexpected output on stderr.
2015-09-08 15:47:34 -04:00
David Steele
adc5568b15 Fixed issue #132: Improved command-line help.
Some platform specific fixes and added regression tests to detect issues.
2015-09-08 12:58:13 -04:00
David Steele
ac3c0d43ab Implemented issue #132: Improved command-line help.
Implemented issue #133: Dynamic module loading where possible.
2015-09-08 07:31:24 -04:00
David Steele
bf3ef15ccc Implemented issue #117: Refactor expiration tests to be purely synthetic 2015-09-01 19:05:10 -04:00
David Steele
6a9377a0a9 Code cleanup and refactoring to standardize on patterns that have evolved over time. 2015-08-29 14:20:46 -04:00
David Steele
d3262822ef Fixed issue #128: Replace config name in test logs
It was actually the bin path that was causing the issue.
2015-08-27 15:55:29 -04:00
David Steele
8c0508789e Implemented issue #25: Option to call stop_backup with every backup. 2015-08-08 17:11:20 -04:00
David Steele
12e8a7572c Worked on issue #122: 9.5 Integration.
Expiration tests worked differently with checkpoint_segments.  Only allow this test < 9.5 until purely synthetic tests are written.
2015-08-08 00:51:58 -04:00
David Steele
2edf5d4bf7 Split most of README.md out into CHANGELOG.md and USERGUIDE.md.
Added release dates to change log.
2015-08-07 14:43:53 -04:00
David Steele
4e7bd4468a Worked on issue #122: 9.5 Integration.
Most tests are working now.  What's not working:

1) --target-resume option fails because pause_on_recovery setting was removed.  Need to implement to the new 9.5 option and make that work with older versions in a consistent way.
2) No tests for the new .partial WAL segments that can be generated on timeline switch.
2015-08-06 16:36:55 -04:00
David Steele
ca1fd9740a Working on issue #117: Refactor expiration tests to be purely synthetic.
Split BackupTest.pm into two modules.  It was getting ungainly to work on.
2015-08-06 00:00:30 -04:00
David Steele
576d50fc1e Added ssh control master config to speed regression testing.
Added vagrant config for Centos 7.
2015-08-05 18:54:41 -04:00
David Steele
23e6267adc Implemented issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 12:32:12 -04:00
David Steele
f8a2da9400 Work on issue #48: Abandon threads and go to processes
* Major refactoring of the protocol layer to support this work.
* Fixed protocol issue that was preventing ssh errors (especially connect) from being logged.
2015-08-05 08:43:41 -04:00
David Steele
1b0f997f59 Implemented issue #119: Make sure all repo permissions are 640/750.
Updated remaining test logs.
2015-07-24 18:05:06 -04:00
David Steele
97c036973c Implemented issue #119: Make sure all repo permissions are 640/750. 2015-07-23 13:11:38 -04:00
David Steele
f3199d13b1 Implement issue #115: When restoring copy pg_control last. 2015-07-16 11:12:48 -04:00
David Steele
65847b5fb1 Vagrant backrest mount is no longer RO so development can be done entirely within VMs. 2015-07-13 13:25:04 -04:00
David Steele
89c0f23f37 Update release notes and some test fixes. 2015-07-13 12:13:07 -04:00
David Steele
073e3b7c00 Minor for regression tests.
Improved CentOS 6 vagrant config.
2015-07-11 22:47:22 -04:00
David Steele
b777525f62 Added --exe param to allow testing of packaged versions.
Updates to u12 vagrant vm.
2015-07-11 17:16:35 -04:00
David Steele
e9ce4efedf Removed dependency on dequeue_timed() which eliminates the CPAN install.
Added vagrant config for CentOS 6.
2015-07-10 09:20:28 -04:00
David Steele
8b7fd99637 Added vagrant configuration for Ubuntu 12.04.
Changes to regression tests to work better with VMs.
2015-07-06 21:00:48 -04:00
David Steele
7248795b91 Work on issue #48: Abandon threads and go to processes
Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.  Using open3 will also be used for local processes so it make sense to switch now.
2015-06-29 22:07:42 -04:00
David Steele
c59adfc68d Shorted test command line params. 2015-06-27 11:12:44 -04:00
David Steele
760baab6b2 Implemented issue #111: Add the psql path for *BSD.
Author: Eric Radman
Modified by Committer
2015-06-22 14:35:25 -04:00
David Steele
dcad0f7007 Merged issue #111: Print the real exception when DBH->connect fails.
Author: Eric Radman
2015-06-22 14:20:56 -04:00
David Steele
f210fe99c3 Implemented issue #109: Move VERSION into source code.
Also stopped replacing FORMAT number which explains the large number of test log changes.  FORMAT should change very rarely and cause test log failures when it does.
2015-06-22 13:11:07 -04:00
David Steele
3f841fcd95 Improved issue #110: 'db-version' is required but not defined.
Improved the error message and added hints.
2015-06-22 09:51:16 -04:00
David Steele
61dfeca3c1 Fixed issue #108: Incompatibility with Perl 5.10.1 2015-06-21 12:06:13 -04:00
David Steele
06c2904212 Change operation constants and functions to command. 2015-06-18 16:55:09 -04:00
David Steele
4e2c14623b Work on issue #48: Abandon threads and go to processes
More separation of the protocol and remote layers than was done in issue #106.
Settings are passed to the remote via command-line parameters rather than in the protocol.
2015-06-18 15:39:30 -04:00
David Steele
22e126eda7 Fix for logic that removes trailing spaces from code. 2015-06-17 15:40:12 -04:00
David Steele
a050b76fa9 Implement issue #107: Better debug commands. 2015-06-17 15:33:58 -04:00
David Steele
9511f9c35c Implemented issue #106: Roll pg_backup_remote into pg_backrest 2015-06-17 11:26:07 -04:00
David Steele
8459e25e32 Test fix for issue #96. 2015-06-14 11:49:55 -04:00
David Steele
b865070edd Experimental 9.5 support. Unit tests are not working yet. 2015-06-14 10:12:36 -04:00
David Steele
0b6f81a812 Implemented issue #50: Reverse ordering by file size. 2015-06-14 09:36:58 -04:00
David Steele
57640d1ac3 Implemented issue #96: Executable names 2015-06-14 08:37:37 -04:00
David Steele
148836fe44 Implemented issue #26: Info command.
* Includes updating the manifest to format 4.  It turns out the manifest and .info files were not very good for providing information.  A format update was required anyway so worked through the backlog of changes that would require a format change.

* Multiple database versions are now supported in the archive.  Does't actually work yet but the structure should be good.

* Tests use more constants now that test logs can catch name regressions.
2015-06-13 18:25:49 -04:00
David Steele
95d62a82d9 Minor log changes for formatting and consistency. 2015-05-31 09:50:42 -04:00
David Steele
200e3b26fe Prevent log test regexp replacement when multithreaded. Logging should be completely disabled in this case. 2015-05-29 22:11:51 -04:00
David Steele
1586e0eb75 Implement issue #92: Check for ERRORs in backup full tests 2015-05-29 17:58:13 -04:00
David Steele
5fd4f3bd5b Improvements to log testing.
Replaced tokens are now numbered to give a more accurate rendition of the logs.
Some formatting improvements.
2015-05-29 14:39:43 -04:00
David Steele
13e4eec629 Implement issue #90: Move archive-max-mb archive-stop checks to end of archive-push. 2015-05-29 11:41:19 -04:00
David Steele
7ace66e544 Implement issue #68: Allow duplicate WAL to be pushed if the checksum matches. 2015-05-29 08:12:39 -04:00
David Steele
6d2f2d3f91 Fix for issue #74: More helpful message when targeting missing stanzas. 2015-05-28 20:52:42 -04:00
David Steele
3083e5fc7e Fix for issue #83: Provide more helpful error message during archive-stop situations.
Fix for issue #84: archive-async in combination with archive-max-mb doesn't work as documented.
Unit tests for archive stop.
2015-05-28 13:17:34 -04:00
David Steele
ca0c866d3b Implement issue #71: log a message when waiting for a checkpoint
Some other log improvements
2015-05-26 12:26:59 -04:00
David Steele
5c43e7035a Implement issue #79: Support comment characters in config file 2015-05-26 11:36:03 -04:00
David Steele
d5335b40e8 Fix for issue #80: enabling archive-copy causes failing differential & incremental backups 2015-05-22 14:49:14 -04:00
David Steele
a7d6382cf2 Unit test fix for absolute path change in a0d985b7ad. 2015-05-08 19:33:34 -04:00
David Steele
9aa41ee76b Added tablespace setting to allow compact database restores for development, staging, etc. 2015-05-08 18:34:27 -04:00
David Steele
a0d985b7ad Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path. 2015-05-07 17:37:25 -05:00
David Steele
1ac4b781fd Better info logging for restore. Most of the messages were debug before and some important ones were missing. 2015-05-07 15:56:56 -06:00
David Steele
095a9a0b83 Added resume setting to allow disabling resume feature. 2015-05-07 10:29:30 -06:00
David Steele
328c2ca5c0 Much better resume:
1) Re-checksums files that have checksums in the manifest
2) Recopies files that do not have a checksum
3) Saves the manifest at regular intervals to preserve checksums
4) Unit tests for all cases (that I can think of)
2015-05-06 18:24:34 -06:00
David Steele
56588f6fdd Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests. 2015-05-05 11:08:48 -06:00
David Steele
a9b9ec2492 v0.61: bug fix for uncompressed remote destination
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination.  The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
2015-04-21 09:45:03 -04:00
David Steele
5185d49873 v0.60: better version support and WAL improvements
* Pushing duplicate WAL now generates an error.  This worked before only if checksums were disabled.

* Database System IDs are used to make sure that all WAL in an archive matches up.  This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.

* Regression tests working back to PostgreSQL 8.3.

* Improved threading model by starting threads early and terminating them late.
2015-04-19 17:53:29 -04:00
David Steele
b37d59832f v0.50: restore and much more
* Added restore functionality.

* All options can now be set on the command-line making pg_backrest.conf optional.

* De/compression is now performed without threads and checksum/size is calculated in stream.  That means file checksums are no longer optional.

* Added option `--no-start-stop` to allow backups when Postgres is shut down.  If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created).  This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.

* Fixed broken checksums and now they work with normal and resumed backups.  Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things.  Issue #28 has been created for checksum deltas.

* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.

* Removed dependency on Moose.  It wasn't being used extensively and makes for longer startup times.

* Checksum for backup.manifest to detect corrupted/modified manifest.

* Link `latest` always points to the last backup.  This has been added for convenience and to make restores simpler.

* More comprehensive unit tests in all areas.
2015-03-25 15:15:55 -04:00
David Steele
4bc4d97f2b v0.30: core restructuring and unit tests
* Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations.  Compression is performed in threads rather than forked processes.

* Fairly comprehensive unit tests for all the basic operations.  More work to be done here for sure, but then there is always more work to be done on unit tests.

* Removed dependency on Storable and replaced with a custom ini file implementation.

* Added much needed documentation (see INSTALL.md).

* Numerous other changes that can only be identified with a diff.
2014-10-05 19:49:30 -04:00
David Steele
587bd1f8d9 v0.10: Backup and archiving are functional
This version has been put into production at Resonate, so it does work, but there are a number of major caveats.

* No restore functionality, but the backup directories are consistent Postgres data directories.  You'll need to either uncompress the files or turn off compression in the backup.  Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.

* Archiving is single-threaded.  This has not posed an issue on our multi-terabyte databases with heavy write volume.  Recommend a large WAL volume or to use the async option with a large volume nearby.

* Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread.  There is an overall process timeout that resolves this issue by killing the process.  Yes, very ugly.

* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes.  Checksums from previous backups are correctly recorded and a full backup will reset everything.

* The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well.  Would definitely like to save these as human-readable text.

* Absolutely no documentation (outside the code).  Well, excepting these release notes.

* Lots of other little things and not so little things.  Much refactoring to follow.
2014-03-05 19:53:13 -05:00