db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Code Issues Releases Activity
1,920 Commits 4 Branches 113 Tags 103 MiB
a029eba8d5
Commit Graph

912 Commits

Author SHA1 Message Date
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get. 
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
 2018-04-30 17:27:39 -04:00
David Steele
321a28f6b0 Add walSegmentNext() and walSegmentRange(). 2018-04-29 11:47:50 -04:00
David Steele
be02c67503 Add pgControlInfo() to read pg_control and determine the PostgreSQL version. 2018-04-29 11:20:51 -04:00
David Steele
8c6e2bdbc7 Add storageInfo() and track size in read objects. 2018-04-29 11:02:21 -04:00
David Steele
d74c167420 Add bufNewC() and only reallocate buffer when the size has changed. 2018-04-29 10:41:05 -04:00
David Steele
d44848baa0 Add strLstExists(), strLstExistsZ(), strSub(), and strSubN() to String and StringList objects. 2018-04-29 10:32:46 -04:00
David Steele
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
Cynthia Shang
721432f6ae Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. 
Contributed by Cynthia Shang.
 2018-04-25 15:46:05 -04:00
Cynthia Shang
8eabc52f3b Add strUpper, strLower, strChr, and strTrunc to String object. 
Contributed by Cynthia Shang.
 2018-04-25 15:16:42 -04:00
David Steele
88a633da17 Divide tests into three types (unit, integration, performance). 
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.

Also finish renaming Perl unit tests with the -perl suffix.
 2018-04-24 09:12:25 -04:00
David Steele
bb8c315cad Storage object improvements. 
* Add storageCopy(), storageMove(), and storagePathSync().
* Separate StorageFile object into separate read and write objects.
* Abstract out Posix file read/write objects.
 2018-04-23 17:26:27 -04:00
David Steele
18882cb882 Add bufEq and bufCat to Buffer object. 2018-04-23 10:19:45 -04:00
David Steele
f146502eae Add TEST_ERROR_FMT macro to simplify testing of formatted error messages. 2018-04-23 10:10:55 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files. 
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
 2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max. 
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
 2018-04-18 19:37:35 -04:00
David Steele
ea4046a607 Try tweaking time sync settings to prevent clock drift rather than restarting VBoxService on every test run. 2018-04-18 18:47:43 -04:00
David Steele
0381945caa Show command parameters as well as command options in initial info message. 2018-04-17 18:47:14 -04:00
David Steele
e9790fd3b1 Update out-of-date description for the spool-path option. 2018-04-17 17:38:25 -04:00
David Steele
cf0ec44ede Rename Perl tests so they don't conflict with their C counterparts. 2018-04-17 12:03:03 -04:00
David Steele
dce11ce3cf Generate code counts for all source files. 
The source files are also classified by type and purpose.
 2018-04-16 15:52:17 -04:00
David Steele
3e22977484 Use pip 9.03 in test VMs. 
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
 2018-04-16 15:25:50 -04:00
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier. 
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
 2018-04-13 16:05:52 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C. 
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
 2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C. 
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
 2018-04-11 09:36:12 -04:00
David Steele
4744eb9387 Add storagePathRemove() and use it in the Perl Posix driver. 
This implementation should be faster because it does not stat each file.  It simply assumes that most directory entries are files so attempts an unlink() first.  If the entry is reported by error codes to be a directory then it attempts an rmdir().
 2018-04-11 08:21:09 -04:00
David Steele
c9ce20d41a Skip version checks when testing in --dev mode. 2018-04-09 16:46:36 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module. 
These are mock integration tests so they should be grouped with the other mock integration tests.
 2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all. 
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
 2018-04-08 12:16:32 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml. 
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
 2018-04-08 10:19:24 -04:00
David Steele
b77b425096 Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave. 2018-04-07 19:44:00 -04:00
David Steele
82751b3b51 Split debug and assert code into separate headers. 
Assert can be used earlier because it only depends on the error-handler and not logging.
 2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad More C type cleanup. 
* Remove typec.h entirely.
* Order all typdefs above local includes.
 2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3 Allow MemContext objects to be copied to a new parent. 
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
 2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4 Cleanup C types. 
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
 2018-04-05 22:38:11 -04:00
David Steele
bd25248df0 Abstract Posix storage driver code into a separate module. 2018-04-05 14:48:16 -04:00
David Steele
348278bb68 Make backup directory sync more efficient. 
Scanning the entire backup directory can be very expensive if there are a lot of small tables. The backup manifest contains the backup directory list so use it to perform syncs instead of scanning the backup directory.
 2018-04-03 21:30:15 -04:00
David Steele
0dce8ba2fa Make backup.history sync more efficient. 
Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.
 2018-04-03 20:54:18 -04:00
David Steele
93fdb98d15 Storage object improvements. 
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
 2018-04-03 12:25:21 -04:00
David Steele
90f980fe91 Remove documentation describing info --output=json as experimental. 2018-03-30 21:33:30 -04:00
David Steele
635caff573 Add ASSERT() that is preserved in production builds. 2018-03-30 19:10:34 -04:00
David Steele
a8721ffe11 Improve logic for smart builds to include version changes. 2018-03-27 22:48:22 -04:00
David Steele
5518a2ac51 Full branch coverage for storage/storage module. 2018-03-27 20:40:33 -04:00
David Steele
c1ab7b3c98 Start work on C handle io object and use it to output help. 2018-03-25 11:39:24 -04:00
David Steele
9001b9b957 Replace THROW_ON_SYS_ERROR() with THROW_SYS_ERROR(). 
The former macro was hiding missing branch coverage for critical error handling.
 2018-03-24 14:11:29 -04:00
David Steele
dabf727d91 Include VM type in gcov path to avoid conflicts between VMs with different architectures. 2018-03-21 16:08:08 -04:00
David Steele
c6457b070f Full branch coverage for config/parse module. 2018-03-21 13:46:08 -04:00
David Steele
b234f43c9d Refactor usec to msec in common/time.c. 
The implementation provides usec resolution but this is not needed in practice and it makes the interface more complicated due to the extra zeros.
 2018-03-21 09:18:48 -04:00
David Steele
31830bdc55 Full branch coverage for all common/type modules. 2018-03-20 16:08:52 -04:00
David Steele
a2030da200 Specify prefix for lcov output. 
This prevents pages from moving around depending on the subtest.
 2018-03-20 16:01:38 -04:00
David Steele
f4ed36d128 Add switch() to lcov branch exclusions. 
lcov requires default to show complete coverage but --Wswitch-enum enforces all enum values be present so no default is needed.

Add documentation for each branch exclusion.
 2018-03-20 10:17:53 -04:00
David Steele
d34142dee8 Update lcov settings. 
Mark any coverage less that 100% as yellow, less that 90% as red.

Add comments to config file and make genhtml use it as well as lcov.
 2018-03-20 09:47:51 -04:00
David Steele
07f38f584a Use lcov for C unit test coverage reporting. 
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.

Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
 2018-03-19 23:33:28 -04:00
David Steele
45c6a0e864 Remove hard-coded release version from loadTest.c. 2018-03-19 21:03:42 -04:00
David Steele
5b0748cf93 Always autogenerate code if --smart is not specified. 2018-03-19 20:25:55 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present. 
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
 2018-03-19 19:54:03 -04:00
David Steele
8e9e97728c Remove --smart from --expect tests. 
This ensures that new binaries are built before running the tests.
 2018-03-19 17:38:44 -04:00
David Steele
2c1687721e Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected. 
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.

Reported by Clinton Adams.
 2018-03-19 13:08:42 -04:00
David Steele
0c313713b1 Use clang for static code analysis during lint testing. 
Nothing found except for some functions that should have been marked __noreturn__.
 2018-03-18 13:32:19 -04:00
David Steele
f0451c1494 Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings. 2018-03-15 13:53:03 -04:00
David Steele
8a1ce42c30 Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line. 
Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem.

Update command documentation to omit secure options that cannot be specified on the command-line.

Reported by Brad Nicholson.
 2018-03-15 12:02:09 -04:00
David Steele
dd3ce70810 Config test code writes secure options to a file instead of passing on the command-line. 2018-03-15 11:09:46 -04:00
David Steele
4fb75c9cc1 Improve conversion of C exceptions to Exception objects. 
Colons in the message would prevent all of the message from being loaded into the Exception object.
 2018-03-15 11:03:28 -04:00
David Steele
4b6cd92a4c Improve debugging. 
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
 2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a Add ASSERT_DEBUG() macro for debugging. 
Replace all current assert() calls except in tests that can't use the debug code.
 2018-03-12 14:31:22 -04:00
Cynthia Shang
dd31ae832d Port most of Config::Config::configLoad() from Perl to C. 
Contributed by Cynthia Shang.
 2018-03-12 11:25:20 -04:00
David Steele
03f1082e86 Improve logging. 
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
 2018-03-12 11:03:38 -04:00
David Steele
94494a632f Update pip before installing awscli. 2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
b987f05f35 Improve bin and libc build performance. 2018-03-09 21:24:30 -05:00
David Steele
c32ee34612 Build with -DNDEBUG by default but disable for testing. 2018-03-09 18:28:34 -05:00
David Steele
fbe4c40386 Move logic for setting Perl configuration defaults to C. 2018-03-09 09:30:50 -05:00
David Steele
919635b6f4 Set config before Main::main() call to avoid secrets being exposed in a stack trace. 2018-03-08 16:24:16 -05:00
Cynthia Shang
34898ce471 Error when multiple arguments are set in the config file for an option that does not accept multiple arguments. 
Contributed by Cynthia Shang.
 2018-03-08 14:15:05 -05:00
David Steele
e331564c90 Add new test for Common::Io::Process. 
Show that output on stderr will raise an exception on close() even if the exit code is 0.
 2018-03-08 12:36:55 -05:00
David Steele
62a9bdbd4c Fix issue passing --no-config to embedded Perl. 
Reported by Ibrahim Edib Kokdemir.
 2018-03-08 10:43:09 -05:00
Cynthia Shang
45ffe37e4f Add cfgOptionTest(). 
Update cfgOption() calls that are better implemented as cfgOptionTest().

Contributed by Cynthia Shang.
 2018-03-05 18:51:48 -05:00
David Steele
c07aa2ae19 Improve code documentation in config module. 2018-03-02 22:42:02 -05:00
David Steele
406acf9cba Enable -Wconversion and silence new warnings. 2018-03-02 12:07:12 -05:00
David Steele
af805af044 Remove Debian package patch now that it has been merged upstream. 2018-03-01 16:59:13 -05:00
David Steele
5d1722b8ef Disable console display of coverage for C files. 
Devel::Cover does not handle it well.
 2018-03-01 13:32:51 -05:00
David Steele
57d70a5649 Remove RHEL package patch since upstream has been updated. 2018-03-01 13:31:03 -05:00
David Steele
5b491b3035 Add CentOS/RHEL package builds. 2018-02-26 19:32:27 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
Cynthia Shang
f75ba7db94 Update backup host to repository host in the documentation. 
Contributed by Cynthia Shang.
 2018-02-19 13:17:58 -05:00
Cynthia Shang
4352407777 Option rename cleanup based on review. 
* Remove --reset tags from v1 options.
* Use constants for repo prefix.
* Specify CFGDEF_INDEX_PG in option structure rather than adding in code.
* Fix error message references to "backup host".

Contributed by Cynthia Shang.
 2018-02-19 10:12:51 -05:00
David Steele
9070325760 Improve performance of HTTPS client. 
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
 2018-02-18 14:53:27 -05:00
David Steele
27678f6188 Clean up and document Makefile. 2018-02-18 10:44:58 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp(). 
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
 2018-02-14 17:19:54 -05:00
David Steele
81952c41f4 Report coverage errors via the console. 
This helps with debugging coverage issues on remote services like Travis.
 2018-02-14 17:13:18 -05:00
David Steele
5f2884cb29 Suppress coverage failures for Archive/Push/Async on Travis. 
The coverage report shows some code as never being run -- but that makes no sense because the tests pass.  This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.

Suppress for now with a plan to implement LCOV for the C unit tests.
 2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d Build C binary/library for C unit tests that require Perl. 2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48 Move Perl option JSON generation to a new module. 
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
 2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5 Generate suppressions when running Valgrind. 
This makes it easier to suppress errors that are only occurring on Travis.
 2018-02-14 09:18:27 -05:00
Cynthia Shang
00e9aca872 Fixed issue where --reset-* was not passing the default value to Perl. 
Contributed by Cynthia Shang.
 2018-02-13 15:58:14 -05:00
David Steele
7c58fe7451 Use 64-bit integers for config options. 
In particular, archive-queue-max requires a greater range than is provided by a 32-bit integer.
 2018-02-09 13:54:33 -05:00
David Steele
305a3e1761 Add int64 variant type. 2018-02-09 13:52:02 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml. 
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
 2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5 Update URL for Debian package repository. 2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
808b6000eb Fix some bugs in C config code: 
1) Error when the same option is defined multiple times in a section using alternate names.
2) Fix logging of invalid command error.
3) Warn when a stanza-only option is in a global section.

Also, make a note to add validation of section names to the check command.

Per review by Cynthia Shang.
 2018-02-05 15:48:26 -05:00
David Steele
d35757c05f Fix missing log-level-stderr option for stanza-* commands. 2018-02-05 15:19:28 -05:00
David Steele
c3f47bf240 Move config parsing out of Perl tests. 
cfgParseTest() is provided in the C library for parsing configs in unit tests.
 2018-02-05 12:32:30 -05:00
David Steele
db21b7a360 Cleanup usage of internal options. 
Apply internal to options that need to be read to determine locality but should not appear in the help.
 2018-02-05 11:50:09 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. 
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
 2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists. 
Contributed by Cynthia Shang.
 2018-02-01 14:12:03 -05:00
David Steele
298b33988c Fix non-compliant formatting for function declarations. 
Per review by Cynthia Shang.
 2018-01-31 18:22:25 -05:00
Cynthia Shang
bacc08792d Ensure latest db-id is selected on when matching archive.info to backup.info. 
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).

Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
 2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include. 
Fixed by Cynthia Shang.
 2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl. 
Options are passed to Perl as JSON from the C process.
 2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05 Improve speed of C unit tests. 
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
 2018-01-26 16:41:17 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
404ec30d03 Move Perl help test to real module. 2018-01-23 12:58:35 -05:00
David Steele
3dd882a79e Add internal flag to config definition. 2018-01-23 12:49:04 -05:00
David Steele
41bd5e8148 Improve KeyValue object. 
Add kvGetList() to get a value as a list (of 1) even if it is a scalar.
 2018-01-23 12:40:02 -05:00
David Steele
2afb73da1c Improve String, StringList, and List objects. 
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
 2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b The archive-push command is now partially coded in C. 
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
 2018-01-17 15:52:00 -05:00
David Steele
1f39a34abc Move lock release later in exitSafe(). 
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
 2018-01-17 10:31:21 -05:00
David Steele
7c73f4f6c9 Add command and exit modules. 2018-01-17 09:15:51 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
eb452c8add Improvements to Storage object. 
* storagePath() and storageList() commands
* Helper functions for local and spool storag
 2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f Improve error module. 
Add functions to convert error codes to C errors and handle system errors.
 2018-01-16 13:29:27 -05:00
David Steele
bffc6c49b3 Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work. 2018-01-16 13:23:08 -05:00
David Steele
d633417e5d Add Wait object and time module. 2018-01-16 13:05:00 -05:00
David Steele
721d1127a0 Add RegExp object. 2018-01-16 12:57:23 -05:00
David Steele
05c731692b Add varStrForce() to Variant object. 2018-01-16 12:54:06 -05:00
David Steele
83ddd75d82 Improvements to String object. 
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
 2018-01-16 12:47:08 -05:00
David Steele
d992851456 Rename Perl tests so they don't conflict with their C counterparts. 2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas. 
Contributed by Cynthia Shang.
 2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
d8c575801d Add Ini object. 2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff Add Storage object. 2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43 Add KeyValue object. 2017-12-22 22:05:37 -05:00
David Steele
2add6cef95 Add Variant and VariantList objects. 2017-12-22 21:50:27 -05:00
David Steele
c4b30af645 Add Buffer object. 2017-12-22 19:56:01 -05:00
David Steele
f89b027dec Improve String and StringList modules. 2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e Run valgrind on all C unit tests. 
Also refactor code to make valgrind happy.
 2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c No longer run master branch through CI. 
The integration branch will be run through CI and then pushed to master with github status checks.
 2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster. 
Fixed by Cynthia Shang.
Reported by Stephen Frost.
 2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10. 
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
 2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built. 
Contributed by Cynthia Shang.
 2017-12-13 11:16:27 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated. 
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
 2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore. 
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
 2017-12-05 14:59:09 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
4f1725dc36 Replace cfgCommandTotal()/cfgOptionTotal() functions with constants. 
The constants are applicable in more cases and allow the compiler to optimize certain loops more efficiently.
 2017-12-05 09:45:51 -05:00
David Steele
59fd92fdd5 Move restore test infrastructure to HostBackup.pm. 
Required to test restores on the backup server, a fairly common scenario.

Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
 2017-12-03 17:08:49 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb Add unit tests for the Manifest module. 
Also minor changes to Manifest module, mostly for test reproducibility.

Contributed by Cynthia Shang.
 2017-11-28 11:44:24 -05:00
David Steele
4a0eb6863d Sync time to prevent build failures when running on VirtualBox. 2017-11-27 22:16:24 -05:00
David Steele
332861c6c9 Only build C binary/library for Perl unit/integration tests. 2017-11-27 21:19:59 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl. 
This allows certain time-critical commands (like async archive-push) to run more quickly.
 2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18 Add List and StringList objects to simplify list handling. 2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b Add String object to simplify string handling. 2017-11-26 17:49:10 -05:00
David Steele
7cf955425e The C library is now required. 
This eliminates conditional loading and eases development of new library features.
 2017-11-26 17:45:00 -05:00
David Steele
c164fc5fb9 Add memGrowRaw() to memory context module. 2017-11-26 12:22:50 -05:00
David Steele
ac54aac448 Fixed an issue passing parameters to remotes. 
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
 2017-11-21 13:17:44 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function. 
This allows the code to be tested more precisely and doesn't require executing a remote process.
 2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0. 
The filter was used with compress level set to 0 which added overhead without any benefit.
 2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore. 
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
 2017-11-20 12:46:35 -05:00
David Steele
8ccfaa9ad3 Designate a single distro (Ubuntu 16.04) for C coverage testing 
Enable optimizations on the distros that don't do coverage testing.
 2017-11-20 09:05:55 -05:00
David Steele
deae0b8cd7 Refactor C page checksum unit test to compile with -Wstrict-aliasing. 2017-11-20 08:58:47 -05:00
David Steele
5d741ea9a9 Refactor C unit test macros so they compile with -Wstrict-aliasing. 2017-11-19 16:46:31 -05:00
David Steele
3b43047e0d Disable -Wclobber compiler warning. 
...because it is mostly useless but keep the rest of of -Wextra.
 2017-11-19 16:44:33 -05:00
David Steele
dc1a5c18ac Wrap try in a do...while loop to make sure that no random else is attached to the main if block. 2017-11-19 16:30:23 -05:00
David Steele
9395ad7043 Improve base64 implementation. 
Different encoded strings could be generated based on compiler optimizations. Even though decoding was still successful the encoded strings did not match the standard.
 2017-11-18 22:49:43 -05:00
David Steele
500d6b4b66 Automated tests for 32-bit i386/i686 architecture. 2017-11-18 20:14:14 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files. 
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
 2017-11-18 20:02:54 -05:00
David Steele
b701cd5a5f Designate a single distro (Ubuntu 16.04) for Perl coverage testing. 
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
 2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions. 
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
 2017-11-16 17:18:51 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test. 
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
 2017-11-15 13:17:57 -05:00
David Steele
22fe684b1f Only check expect logs on CentOS 7. 
Variations in distros cause false negatives in tests but don't add much value.
 2017-11-14 17:16:39 -05:00
David Steele
b8746f368d Inflate performance improvement for gzip filter and full unit test coverage. 2017-11-14 15:12:31 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore. 
Reported by Craig A. James.
 2017-11-14 14:48:44 -05:00
David Steele
cf7da546a3 Simplify try..catch..finally names. 2017-11-13 21:22:13 -05:00
David Steele
bcdfc7d0b5 Enable additional warnings for C builds. 2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a Repository encryption support. 
Contributed by Cynthia Shang.
 2017-11-06 12:51:12 -05:00
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
55f68ef013 Remove configurable option hints. 
db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.
 2017-10-30 10:58:04 -04:00
David Steele
1ef27ec8c2 Add list type for options. 
The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.
 2017-10-30 10:50:35 -04:00
David Steele
f57e376c44 Update Debian/Ubuntu containers to download lastest version of pip. 2017-10-30 09:29:46 -04:00
David Steele
8674a4f7ae Allow functions with sensitive options to be logged at debug level with redactions. 
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
 2017-10-24 12:35:36 -04:00
David Steele
b2cc65d886 Improve performance of test code. 
Wait when all tests have been assigned to reduce CPU load.
 2017-10-24 10:35:42 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored. 
Reported by Jens Wilke.
 2017-10-22 18:05:46 -04:00
David Steele
3553ccae44 Add I/O performance tests. 2017-10-22 13:27:47 -04:00
David Steele
9b98c2e7d6 Run all tests on tempfs rather than local disk. 2017-10-22 13:11:03 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
44c7c0475a Improve protocol error handling. 
In particular, "stop" errors are no longer reported as "unexpected".
 2017-10-21 14:11:33 -04:00
David Steele
f4524aeaa9 Add base64 encode/decode. 2017-10-18 11:14:26 -04:00
Cynthia Shang
2d56de00fc Fix instances where database-id was not rendered as an integer in JSON info output. 
Fixed by Cynthia Shang.
Reported by Jason O'Donnell.
 2017-10-18 08:42:32 -04:00
David Steele
b1efd598cf Add PostgreSQL versions to Debian VMs for testing. 2017-10-16 18:21:47 -04:00
David Steele
c94e52f697 Remove error when overlapping timelines are detected. 
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.

Reported by blogh.
 2017-10-16 14:34:44 -04:00
David Steele
583a76f605 Add C memory contexts. 2017-10-16 11:25:49 -04:00
David Steele
6f5186f9e6 Convert config tests into C unit tests. 2017-10-16 11:13:51 -04:00
David Steele
9d78948a14 Improvements to page checksum module and tests: 
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
 2017-10-16 11:03:06 -04:00
David Steele
80758f0023 Add test macros for C results and errors. 2017-10-16 10:54:35 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler. 
Basic try ... catch ... finally pattern to simplify error handling in C.
 2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5 Add C unit test infrastructure. 2017-10-12 12:55:48 -04:00
David Steele
abea4d1fd5 Remove Debian test repo after PostgreSQL 10 release. 2017-10-05 12:33:13 -04:00
David Steele
d89b7d5862 Warnings in C builds treated as errors. 2017-09-30 10:48:25 -04:00
David Steele
40a85e34f1 Improve C library smart build. 
Ignore changes outside of /lib/pgBackRest/Config.
 2017-09-20 19:57:05 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
7c95a2f4de Updates to installation documentation. 
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
 2017-09-08 21:50:00 -07:00
David Steele
6343fdd584 Additional backup exclusions. 
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
 2017-09-04 08:26:57 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. 
Suggested by Victor Gdalevich.
 2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.` 
Contributed by Cynthia Shang.
 2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files. 
Reported by Jesper St John, Aleksandr Rogozin.
 2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process. 
Reported by William Cox.
 2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
f8de73674c Updated Docker build in Vagrantfile. 2017-08-10 20:52:29 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option 
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
 2017-08-08 18:09:07 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test. 
Contributed by Cynthia Shang.
 2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. 
Fixed by Cynthia Shang.
 2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. 
Contributed by Cynthia Shang.
 2017-08-01 10:34:53 -04:00
David Steele
fa992433da Update CI code to prevent deprecated VM from being rewritten. 2017-07-27 11:31:42 -04:00
David Steele
14d6974d1b Use Google DNS in test environment for consistency. 2017-07-26 13:40:09 -04:00
David Steele
df7d633683 Improvements to command/command-line help: 
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
 2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
David Steele
cbb749e12a Debian/Ubuntu documentation now builds on Ubuntu 16. 2017-07-24 21:12:05 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups. 
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
 2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command. 
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
 2017-07-21 17:32:41 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. 
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
 2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements: 
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
 2017-06-24 10:59:00 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options. 
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
 2017-06-22 18:22:49 -04:00
David Steele
6327799c9a Use old trusty image until new image works with RHEL6. 2017-06-22 18:13:59 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00
David Steele
f8849d7359 Remove patch directory before Debian package builds. 
Patches should be merged into master before trying a new build after a release with patches.
 2017-06-22 12:37:21 -04:00
David Steele
8b9965abcb Remove process-max option. 
Parallelism is now tested in a more targeted manner and the high level option is no longer needed.
 2017-06-21 19:21:16 -04:00
David Steele
2348693389 Reduce process redundancy in mock tests. 2017-06-21 18:47:25 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
73274fc608 Remove Ubuntu 12.04 from CI now that it is EOL (it can still be run manually). 2017-06-21 17:02:01 -04:00
David Steele
c69baa2de1 Install sudo in base containers rather than on demand. 2017-06-21 16:51:41 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
edada21866 Rename archive-push unit test. 
This is part of the archive consolidation but is committed separately so that git will follow the rename.
 2017-06-21 08:45:58 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor. 
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
 2017-06-15 15:32:10 -04:00
David Steele
269ae4556d Fix timeouts in ExecuteTest to speed multi-process testing. 2017-06-15 15:20:02 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor: 
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
 2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process. 
Fixed by Cynthia Shang.
 2017-05-16 11:28:51 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. 
Fixed by Cynthia Shang.
 2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version. 
Fixed by Cynthia Shang.
 2017-05-12 15:49:14 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. 
Suggested by Yogesh Sharma.
 2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid. 
Suggested by Yogesh Sharma.
 2017-04-12 09:53:07 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files. 
In both cases the files are checksummed on resume so there's no danger of partial copies.
 2017-04-12 09:05:57 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings. 
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
 2017-04-10 17:23:39 -04:00
David Steele
8546571461 Ignore clock skew in container libc/package builds using make. 
It is common for containers to have clock skew so the build process takes care of this issue independently.
 2017-04-10 17:20:21 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages. 
Contributed by Cynthia Shang.
 2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm. 
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
 2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers. 
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
 2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring. 
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
 2017-04-07 10:31:49 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade. 
Contributed by Cynthia Shang.
 2017-04-06 19:36:37 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash. 
Refactor File->list() and fileList() to accept optional parameters.
 2017-04-04 21:17:19 -04:00
David Steele
9eae36a702 Fixed flapping archive stop tests. 2017-04-04 21:15:22 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command. 
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
 2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. 
Contributed by Cynthia Shang.
 2017-03-22 19:03:28 -04:00
David Steele
cbd879a983 Updated vagrant to new version and image. 2017-03-14 22:05:17 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. 
Reported by Stephen Frost.
 2017-03-13 15:50:13 -04:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. 
Reported by blogh.
 2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements: 
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
 2017-03-02 11:47:23 -05:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment. 
Reported by Stephen Frost.
 2017-03-01 14:15:46 -05:00
David Steele
774b243719 Fixed dev version string in Debian changelog. 2017-03-01 12:22:47 -05:00
David Steele
2c48cf8d0d Fixed Vagrantfile to match user/group changes made for Travis CI. 2017-03-01 09:36:33 -05:00
David Steele
8161ab1c50 Temporarily disable flapping keep-alive test. 2017-02-26 12:58:48 -05:00
David Steele
abe0935ea5 Disable CI config writing with —no-ci-config. 2017-02-26 12:53:41 -05:00
David Steele
ac485cc355 Change —retry default to 0. 2017-02-22 18:43:10 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
023578c008 Make package tests work with dev builds. 2017-02-21 14:22:03 -05:00
David Steele
0e9bc513c9 Suppress lint on all but the first test. 2017-02-21 12:54:27 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest(). 
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
 2017-02-21 11:56:55 -05:00
David Steele
01312f6260 Improved formatting of testResult() output. 2017-02-21 11:47:45 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
56543f7a99 Added package builds to test suite and other improvements: 
* Automated builds of Debian packages for all supported distributions.
* Added --dev option to aggregate commonly used dev options.
* Added --no-package option to skip package builds.
* C library and packages are built by default, added -smart option to rebuild only when file changes are detected.
* The --libc-only option has been changed to --build-only now that packages builds have been added.
 2017-02-17 22:31:16 -05:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail. 
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
 2017-02-13 19:59:14 -05:00
David Steele
a54018d573 Improved performance of wal archive min/max provided by the info command. 
Suggested by Jens Wilke.
 2017-02-13 10:03:02 -05:00
David Steele
bc2bb040c5 Fixed an issue where an archive-push error would not be retried. 
It would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted).

Reported by Jens Wilke.
 2017-02-13 09:57:10 -05:00
David Steele
498f52da09 Improvements to documentation engine: 
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
 2017-02-10 10:22:05 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option. 
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
 2017-02-05 10:59:49 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
1666ee177f Fixed --no-online tests to suppress expected errors. 2017-02-02 20:44:42 -05:00
David Steele
7418559a67 Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination. 2017-01-31 10:36:59 -05:00
David Steele
eb1a1e5308 Missed expect logs from commit 455c604. 2017-01-30 16:22:12 -05:00
David Steele
34e4cfaeec Fixed minor typos. 2017-01-30 15:48:10 -05:00
David Steele
455c6048ee C Library builds immune to clock differences between VM and host. 2017-01-30 14:00:40 -05:00
David Steele
6b99d770b5 Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN. 2017-01-30 13:59:00 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
18db178ac3 Various improvements to the test suite: 
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
 2017-01-27 09:42:30 -05:00
David Steele
7d1347cdec Split the archive-get and archive-push commands into separate modules. 2017-01-10 19:59:32 -05:00
David Steele
b58045f575 Moved the Archive modules to the Archive directory. 2017-01-10 19:51:20 -05:00
David Steele
2ab63f27b2 C Library builds only run when C library has actually changed. 2017-01-10 13:12:53 -05:00
David Steele
0b4055e588 Moved File->manifest() into the FileCommon.pm module. 2017-01-10 13:08:03 -05:00
David Steele
0e4f51c271 Added integration for testing coverage with Devel::Cover. 2017-01-09 20:49:04 -05:00
David Steele
39744a4f1d Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner(). 
Reported by Leonardo Avellar.
 2017-01-05 15:19:56 -05:00
David Steele
4ff2714bf0 Return proper error code when unable to convert a relative path to an absolute path. 
Suggested by Yogesh Sharma.
 2017-01-04 18:39:50 -05:00
David Steele
c8dfc673b8 Moved tests into new modules to better reflect actual usage. 2017-01-04 18:31:16 -05:00
David Steele
e3d63c0252 Allow multiple --module, --test, and --run options to be used for test.pl. 2017-01-04 18:29:13 -05:00
Cynthia Shang
be337c951a Improvements to stanza-create: 
* Force only required when a change is needed for backup.info or archive.info.
* Identical files are not overwritten.
 2017-01-04 10:12:29 -05:00
David Steele
b91ca11329 Added expect log expression to replace year subdirectories in backup.history. 2017-01-04 10:09:21 -05:00
David Steele
83beab7ec3 Refactor File module to improve test coverage. 
Added unit tests for low-level functions in the BackupCommon module.
 2016-12-23 11:43:26 -05:00
David Steele
5d3473b52d Refactor File module to improve test coverage. 2016-12-23 08:30:34 -05:00
David Steele
6b2666a9d7 Split test modules into separate files. 
Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
 2016-12-23 08:22:59 -05:00
Cynthia Shang
56144c99c0 Improved check command to run on a standby. 
Only basic checks are done because pg_switch_xlog() cannot be executed on a replica.

Contributed by Cynthia Shang.
 2016-12-21 14:35:20 -05:00
Cynthia Shang
974a02dbe4 Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. 
Contributed by Cynthia Shang.
 2016-12-20 16:52:20 -05:00
David Steele
36a5349b1c Added the --checksum-page option. 
This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
 2016-12-12 18:54:07 -05:00
David Steele
6dd0829033 File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process. 2016-12-12 18:47:17 -05:00
David Steele
b4884e5a0b Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database. 
Reported by Michael Vitale.
 2016-12-10 14:05:28 -05:00
David Steele
e61ef6ebce Add more realistic data files to synthetic backup and restore tests. 2016-12-10 09:13:55 -05:00
David Steele
eff7b46eb1 Various improvements to validation of backup and restore. 2016-12-10 09:11:12 -05:00
David Steele
f818901c0e Added the --repo-sync option to allow directory syncs in the repository to be disabled. 
This is for file systems that do not support them, e.g. NTFS
 2016-12-10 09:06:45 -05:00
David Steele
896984848b Improved errors returned from child processes by removing redundant error level and code. 2016-12-04 17:51:00 -05:00
David Steele
817592837c Added a predictable log entry to signal that a command has completed successfully. 
For example a backup ends successfully with: INFO: backup command end: completed successfully.
 2016-12-04 14:44:53 -05:00
David Steele
f823a25ac2 Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error. 
* The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error.
* Removed erroneous --no-config option in help test module.
* Changed the --no-fork test option to --fork with negation to match all other boolean parameters.
 2016-12-04 09:25:40 -05:00
David Steele
d18cd7ad0f Fixed an issue that prevented errors from being output to the console before the logging system was initialized. 
That is, while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not.
 2016-12-03 18:01:17 -05:00
David Steele
2e01e6b9e1 Improved consistency and flexibility of the protocol layer by using JSON for all messages. 2016-12-03 17:34:51 -05:00
Cynthia Shang
b7f1738200 Code cleanup in preparation for improved stanza-create command. 
Contributed by Cynthia Shang.
 2016-12-03 10:28:08 -05:00
David Steele
005fc08795 Added the --repo-link option. 
Allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
 2016-12-03 10:23:26 -05:00
David Steele
6ae4d7de81 Fixed a bug where internal symlinks were not being created for tablespaces in the repository. 
This issue was only apparent when trying to bring up clusters in-place with filesystem snapshots and did not affect normal backup and restore.
 2016-12-03 09:56:05 -05:00
Cynthia Shang
22cd29771f Update control and wal test files to 9.4 with matching system identifiers. 
Contributed by Cynthia Shang.
 2016-12-01 09:33:14 -05:00
David Steele
a850335015 Simplified the result hash of File->manifest(), Db->tablespaceMapGet(), and Db->databaseMapGet(). 2016-11-30 14:36:39 -05:00
David Steele
09df07efb7 For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process. 
The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
 2016-11-30 14:23:05 -05:00
Cynthia Shang
e951c4f9c2 Abstracted code to determine which database cluster is the master and which are standbys. 
Contributed by Cynthia Shang.
 2016-11-30 14:15:11 -05:00
David Steele
d101e9c84e Improved exception handling in file unit tests. 2016-11-30 14:03:58 -05:00
David Steele
c9b49b0d7e Doc engine improvements. 
Bug Fixes:

* Fixed and issue that suppressed exceptions in PDF builds.

Features:

* Allow a source to be included as a section so large documents can be broken up.
* Added section link support to Markdown output.
* Added list support to PDF output.
* Added include option to explicitly build sources (complements the exclude option though both cannot be used in the same invocation).
* Added keyword-add option to add keywords without overriding the default keyword.
* Added debug option to doc.pl to easily add the debug keyword to documentation builds.
* Added pre option to doc.pl to easily add the pre keyword to documentation builds.

Refactoring:

* Improvements to markdown rendering.
* Remove code dependency on project variable, instead use title param.
 2016-11-17 16:35:11 -05:00
David Steele
dbb9d80dab Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches. 
This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode.

Reported by Stephen Frost.
 2016-11-17 09:39:21 -05:00
David Steele
9d9c438f59 Doc engine improvements. 
Bug Fixes:

* Fixed missing variable replacements.
* Removed hard-coded host names from configuration file paths.

Documentation Features:

* Allow command-line length to be configured using cmd-line-len param.
* Added compact param to allow CSS to be embedded in HTML file.
* Added pretty param to produce HTML with proper indenting.
* Only generate HTML menu when required and don't require index page.
* Assign numbers to sections by default.
* VM mount points are now optional.
 2016-11-07 17:06:35 +02:00
David Steele
dd621081b9 Fixed an issue where tablespace paths with the same prefix would cause an invalid link error. 
Reported by Nikhilchandra Kulkarni.
 2016-11-07 16:37:16 +02:00
David Steele
21ca412702 Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed. 2016-11-05 19:22:30 +02:00
David Steele
f43e5bc52d Removed extraneous use lib directives from Perl modules. 
Suggested by Devrim Gündüz.
 2016-11-04 13:56:26 +02:00
David Steele
171c5835bb Another expect log update for stanza-create feature. 2016-11-03 17:48:59 +02:00
David Steele
26f76762b5 Expect log updates for stanza-create feature. 2016-10-17 10:03:42 +01:00
Cynthia Shang
8d1880ca4a Added the stanza-create command to formalize creation of stanzas in the repository. 
Contributed by Cynthia Shang.
 2016-10-14 12:21:47 +01:00
David Steele
16d0e767ef Set application_name to "pgBackRest [command]" for database connections. 
Suggested by Jens Wilke.
 2016-10-05 21:15:10 -04:00
David Steele
0a4099fff8 Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock. 
Suggested by Jens Wilke.
 2016-10-05 21:13:29 -04:00
David Steele
5dc801d59d Fixed an issue where the async archiver would not be started if archive-push did not have enough space to queue a new WAL segment. 
This meant that the queue would never be cleared without manual intervention (such as calling archive-push directly). PostgreSQL now receives errors when there is not enough space to store new WAL segments but the async process will still be started so that space is eventually freed.

Reported by Jens Wilke.
 2016-10-05 09:47:34 -04:00
David Steele
2a18674111 Added the log-level-stderr option. 
Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior.

Suggested by Sascha Biberhofer.
 2016-10-05 09:09:30 -04:00
David Steele
d81198e65e Added the cmd-ssh to allow the ssh client to be specified. 
Suggested by Jens Wilke.
 2016-10-05 08:55:54 -04:00
David Steele
6f34fbc695 Vagrant build improvements: 
* Miminize TeXLive package list to save time during VM builds.
* General cleanup and added timestamps.
* Pin to specific box version.
 2016-10-03 09:15:29 -04:00
David Steele
1491862716 Regression tests can now be run as any properly-configured user, not just vagrant. 2016-10-01 13:39:44 -04:00
David Steele
156859c5d5 Update CentOS/Debian package definitions. 2016-09-29 19:07:46 -04:00
David Steele
3e4ca153ce Fixed broken vm-max option in test.pl. 2016-09-29 18:26:32 -04:00
David Steele
5d033d028c Archive validation improvements: 
* Fixed error message to properly display the archive command when an invalid archive command is detected.
* Check that archive_mode is enabled when archive-check option enabled.
 2016-09-28 19:45:33 -04:00
David Steele
25f40f9b15 Log improvements: 
* pgBackRest version number included in command start INFO log output.
* Process ID logged for local process start/stop INFO log output.
* Fixed missing expect output for help module.
 2016-09-17 11:53:28 -04:00
David Steele
f54145c0cc Fixed timeout issues. 
* Fixed an issue where local processes were not disconnecting when complete and could later timeout. (Reported by Todd Vernick.)
* Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive. (Reported by Todd Vernick.)
 2016-09-14 16:37:07 -05:00
David Steele
e08b56a4e3 Show Process ID in log output instead of filtering it out with the timestamp. 2016-09-14 16:30:57 -05:00
John K. Harvey
f253a4d85c Suppress "dpkg-reconfigure: unable to re-open stdin: No file or directory" warning in Vagrant VM build. 
Contributed by John Harvey.
 2016-09-08 17:29:26 -04:00
Cynthia Shang
d2add293c7 Archive retention improvements: 
* Fixed an issue where retention-archive was not automatically being set when retention-archive-type=diff, resulting in a less aggressive than intended expiration of archive.
* Additional warnings when archive retention settings may not have the intended effect or would allow indefinite retention.
* Closed #235: "Retention policy question" by adding documentation for archive retention.

Contributed by Cynthia Shang.
 2016-09-07 08:07:37 -04:00
David Steele
cc9c8819ee Make uid/gid selection for containers dynamic. 2016-09-06 14:01:39 -04:00
David Steele
cedfcbd4be Update Vagrantfile: 
* Change box provider to bento (hoping for better stability with Docker).
* Install texlive from Debian packages.
 2016-09-06 11:02:01 -04:00
David Steele
dd8d781217 Safely check eval blocks and convert $@ to $EVAL_ERROR. 2016-09-06 09:44:50 -04:00
David Steele
a701309453 Converted Perl threads to processes. 2016-09-06 09:35:02 -04:00
David Steele
4026ad649d Refactor of protocol minions in preparation for the new local minion. 2016-09-06 09:15:22 -04:00
David Steele
f71073c11f Improved temp file handling for archiving. 2016-09-04 19:29:30 -04:00
David Steele
8f61473955 Improve ASSERT error handling. 2016-09-04 09:25:20 -04:00
David Steele
f986e6ae33 File module improvements. 
* Remove obsolete thread index variable from File() module.
* Changed temporary file names to consistently use .pgbackrest.tmp.
 2016-09-04 09:22:53 -04:00
David Steele
bcdb5cdac8 Fixed a issue where tablespaces were copied from the master during standby backup. 2016-09-04 09:19:44 -04:00
David Steele
2feaaf225e Exclude contents of $PGDATA/pg_replslot directory. 2016-09-04 09:13:13 -04:00
Cynthia Shang
634f5c6829 Closed #250: Fixed the check command so backup info is checked remotely and not just locally. 
Fixed by Cynthia Shang.
 2016-08-29 14:09:37 -04:00
David Steele
c9bdf6a9ae Fixed timeout issue. 
The protocol-timeout option was not automatically increased when the db-timeout option was increased.

Reported by Todd Vernick.
 2016-08-25 11:27:00 -04:00
David Steele
5ada189a92 Backup from a standby cluster. 
A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
 2016-08-25 11:25:46 -04:00
David Steele
d0b6f78b20 More flexible configuration for databases 
Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
 2016-08-24 12:39:27 -04:00
David Steele
cd6278e5af Revert some backup exclusions until they have been tested more thoroughly. 2016-08-24 12:27:48 -04:00
David Steele
9c13ca1d09 mprove backup log messages to indicated which host the files are being copied from. 2016-08-24 12:01:08 -04:00
David Steele
f1412baccf Exclude directories during backup that are cleaned, recreated, or zeroed by PostgreSQL at startup. 
These include (depending on the version where they were introduced): pgsql_tmp, pg_dynshmem, pg_notify, pg_replslot, pg_serial, pg_snapshots, pg_stat_tmp, pg_subtrans. The postgresql.auto.conf.tmp file is now excluded in addition to files that were already excluded: backup_label.old, postmaster.opts, postmaster.pid, recovery.conf, recovery.done.
 2016-08-16 09:35:16 -04:00
David Steele
b25f10949a Fixed #229: Check fails on target db host. 
Options that should not allow multiple values could be specified multiple times in pgbackrest.conf without an error being raised.
 2016-08-15 20:15:17 -04:00
David Steele
1e0f15f425 Improve error message for links that reference links in manifest build. 2016-08-15 17:23:37 -04:00
David Steele
f9fa1270b2 Fixed #236: Recursive user tablespace symlink. 
A tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely.
 2016-08-15 17:11:45 -04:00
David Steele
17b79d6279 Database version refactoring. 
* Refactor db version constants into a separate module.
* Update synthetic backup tests to PostgreSQL 9.4.
 2016-08-11 22:35:24 -04:00