db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Code Issues Releases Activity
2,067 Commits 4 Branches 113 Tags 103 MiB
a7d64bab7a
Commit Graph

458 Commits

Author SHA1 Message Date
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier. 
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
 2018-04-13 16:05:52 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C. 
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
 2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C. 
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
 2018-04-11 09:36:12 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module. 
These are mock integration tests so they should be grouped with the other mock integration tests.
 2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all. 
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
 2018-04-08 12:16:32 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml. 
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
 2018-04-08 10:19:24 -04:00
David Steele
b77b425096 Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave. 2018-04-07 19:44:00 -04:00
David Steele
82751b3b51 Split debug and assert code into separate headers. 
Assert can be used earlier because it only depends on the error-handler and not logging.
 2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad More C type cleanup. 
* Remove typec.h entirely.
* Order all typdefs above local includes.
 2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3 Allow MemContext objects to be copied to a new parent. 
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
 2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4 Cleanup C types. 
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
 2018-04-05 22:38:11 -04:00
David Steele
bd25248df0 Abstract Posix storage driver code into a separate module. 2018-04-05 14:48:16 -04:00
David Steele
93fdb98d15 Storage object improvements. 
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
 2018-04-03 12:25:21 -04:00
David Steele
635caff573 Add ASSERT() that is preserved in production builds. 2018-03-30 19:10:34 -04:00
David Steele
c1ab7b3c98 Start work on C handle io object and use it to output help. 2018-03-25 11:39:24 -04:00
David Steele
dabf727d91 Include VM type in gcov path to avoid conflicts between VMs with different architectures. 2018-03-21 16:08:08 -04:00
David Steele
07f38f584a Use lcov for C unit test coverage reporting. 
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.

Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
 2018-03-19 23:33:28 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present. 
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
 2018-03-19 19:54:03 -04:00
David Steele
0c313713b1 Use clang for static code analysis during lint testing. 
Nothing found except for some functions that should have been marked __noreturn__.
 2018-03-18 13:32:19 -04:00
David Steele
f0451c1494 Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings. 2018-03-15 13:53:03 -04:00
David Steele
dd3ce70810 Config test code writes secure options to a file instead of passing on the command-line. 2018-03-15 11:09:46 -04:00
David Steele
4b6cd92a4c Improve debugging. 
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
 2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a Add ASSERT_DEBUG() macro for debugging. 
Replace all current assert() calls except in tests that can't use the debug code.
 2018-03-12 14:31:22 -04:00
David Steele
03f1082e86 Improve logging. 
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
 2018-03-12 11:03:38 -04:00
David Steele
94494a632f Update pip before installing awscli. 2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
919635b6f4 Set config before Main::main() call to avoid secrets being exposed in a stack trace. 2018-03-08 16:24:16 -05:00
David Steele
e331564c90 Add new test for Common::Io::Process. 
Show that output on stderr will raise an exception on close() even if the exit code is 0.
 2018-03-08 12:36:55 -05:00
Cynthia Shang
45ffe37e4f Add cfgOptionTest(). 
Update cfgOption() calls that are better implemented as cfgOptionTest().

Contributed by Cynthia Shang.
 2018-03-05 18:51:48 -05:00
David Steele
406acf9cba Enable -Wconversion and silence new warnings. 2018-03-02 12:07:12 -05:00
David Steele
5b491b3035 Add CentOS/RHEL package builds. 2018-02-26 19:32:27 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
David Steele
9070325760 Improve performance of HTTPS client. 
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
 2018-02-18 14:53:27 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp(). 
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
 2018-02-14 17:19:54 -05:00
David Steele
5f2884cb29 Suppress coverage failures for Archive/Push/Async on Travis. 
The coverage report shows some code as never being run -- but that makes no sense because the tests pass.  This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.

Suppress for now with a plan to implement LCOV for the C unit tests.
 2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d Build C binary/library for C unit tests that require Perl. 2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48 Move Perl option JSON generation to a new module. 
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
 2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5 Generate suppressions when running Valgrind. 
This makes it easier to suppress errors that are only occurring on Travis.
 2018-02-14 09:18:27 -05:00
David Steele
305a3e1761 Add int64 variant type. 2018-02-09 13:52:02 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml. 
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
 2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5 Update URL for Debian package repository. 2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
c3f47bf240 Move config parsing out of Perl tests. 
cfgParseTest() is provided in the C library for parsing configs in unit tests.
 2018-02-05 12:32:30 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. 
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
 2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists. 
Contributed by Cynthia Shang.
 2018-02-01 14:12:03 -05:00
Cynthia Shang
bacc08792d Ensure latest db-id is selected on when matching archive.info to backup.info. 
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).

Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
 2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include. 
Fixed by Cynthia Shang.
 2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl. 
Options are passed to Perl as JSON from the C process.
 2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05 Improve speed of C unit tests. 
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
 2018-01-26 16:41:17 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
404ec30d03 Move Perl help test to real module. 2018-01-23 12:58:35 -05:00
David Steele
2afb73da1c Improve String, StringList, and List objects. 
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
 2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b The archive-push command is now partially coded in C. 
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
 2018-01-17 15:52:00 -05:00
David Steele
7c73f4f6c9 Add command and exit modules. 2018-01-17 09:15:51 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
eb452c8add Improvements to Storage object. 
* storagePath() and storageList() commands
* Helper functions for local and spool storag
 2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f Improve error module. 
Add functions to convert error codes to C errors and handle system errors.
 2018-01-16 13:29:27 -05:00
David Steele
d633417e5d Add Wait object and time module. 2018-01-16 13:05:00 -05:00
David Steele
721d1127a0 Add RegExp object. 2018-01-16 12:57:23 -05:00
David Steele
83ddd75d82 Improvements to String object. 
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
 2018-01-16 12:47:08 -05:00
David Steele
d992851456 Rename Perl tests so they don't conflict with their C counterparts. 2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas. 
Contributed by Cynthia Shang.
 2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
d8c575801d Add Ini object. 2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff Add Storage object. 2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43 Add KeyValue object. 2017-12-22 22:05:37 -05:00
David Steele
2add6cef95 Add Variant and VariantList objects. 2017-12-22 21:50:27 -05:00
David Steele
c4b30af645 Add Buffer object. 2017-12-22 19:56:01 -05:00
David Steele
f89b027dec Improve String and StringList modules. 2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e Run valgrind on all C unit tests. 
Also refactor code to make valgrind happy.
 2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c No longer run master branch through CI. 
The integration branch will be run through CI and then pushed to master with github status checks.
 2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster. 
Fixed by Cynthia Shang.
Reported by Stephen Frost.
 2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10. 
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
 2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built. 
Contributed by Cynthia Shang.
 2017-12-13 11:16:27 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated. 
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
 2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore. 
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
 2017-12-05 14:59:09 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
59fd92fdd5 Move restore test infrastructure to HostBackup.pm. 
Required to test restores on the backup server, a fairly common scenario.

Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
 2017-12-03 17:08:49 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb Add unit tests for the Manifest module. 
Also minor changes to Manifest module, mostly for test reproducibility.

Contributed by Cynthia Shang.
 2017-11-28 11:44:24 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl. 
This allows certain time-critical commands (like async archive-push) to run more quickly.
 2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18 Add List and StringList objects to simplify list handling. 2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b Add String object to simplify string handling. 2017-11-26 17:49:10 -05:00
David Steele
7cf955425e The C library is now required. 
This eliminates conditional loading and eases development of new library features.
 2017-11-26 17:45:00 -05:00
David Steele
ac54aac448 Fixed an issue passing parameters to remotes. 
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
 2017-11-21 13:17:44 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function. 
This allows the code to be tested more precisely and doesn't require executing a remote process.
 2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0. 
The filter was used with compress level set to 0 which added overhead without any benefit.
 2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore. 
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
 2017-11-20 12:46:35 -05:00
David Steele
8ccfaa9ad3 Designate a single distro (Ubuntu 16.04) for C coverage testing 
Enable optimizations on the distros that don't do coverage testing.
 2017-11-20 09:05:55 -05:00
David Steele
3b43047e0d Disable -Wclobber compiler warning. 
...because it is mostly useless but keep the rest of of -Wextra.
 2017-11-19 16:44:33 -05:00
David Steele
500d6b4b66 Automated tests for 32-bit i386/i686 architecture. 2017-11-18 20:14:14 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files. 
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
 2017-11-18 20:02:54 -05:00
David Steele
b701cd5a5f Designate a single distro (Ubuntu 16.04) for Perl coverage testing. 
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
 2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions. 
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
 2017-11-16 17:18:51 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test. 
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
 2017-11-15 13:17:57 -05:00
David Steele
22fe684b1f Only check expect logs on CentOS 7. 
Variations in distros cause false negatives in tests but don't add much value.
 2017-11-14 17:16:39 -05:00
David Steele
b8746f368d Inflate performance improvement for gzip filter and full unit test coverage. 2017-11-14 15:12:31 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore. 
Reported by Craig A. James.
 2017-11-14 14:48:44 -05:00
David Steele
bcdfc7d0b5 Enable additional warnings for C builds. 2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a Repository encryption support. 
Contributed by Cynthia Shang.
 2017-11-06 12:51:12 -05:00
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
f57e376c44 Update Debian/Ubuntu containers to download lastest version of pip. 2017-10-30 09:29:46 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored. 
Reported by Jens Wilke.
 2017-10-22 18:05:46 -04:00
David Steele
3553ccae44 Add I/O performance tests. 2017-10-22 13:27:47 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
f4524aeaa9 Add base64 encode/decode. 2017-10-18 11:14:26 -04:00
David Steele
b1efd598cf Add PostgreSQL versions to Debian VMs for testing. 2017-10-16 18:21:47 -04:00
David Steele
c94e52f697 Remove error when overlapping timelines are detected. 
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.

Reported by blogh.
 2017-10-16 14:34:44 -04:00
David Steele
583a76f605 Add C memory contexts. 2017-10-16 11:25:49 -04:00
David Steele
6f5186f9e6 Convert config tests into C unit tests. 2017-10-16 11:13:51 -04:00
David Steele
9d78948a14 Improvements to page checksum module and tests: 
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
 2017-10-16 11:03:06 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler. 
Basic try ... catch ... finally pattern to simplify error handling in C.
 2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5 Add C unit test infrastructure. 2017-10-12 12:55:48 -04:00
David Steele
abea4d1fd5 Remove Debian test repo after PostgreSQL 10 release. 2017-10-05 12:33:13 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
7c95a2f4de Updates to installation documentation. 
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
 2017-09-08 21:50:00 -07:00
David Steele
6343fdd584 Additional backup exclusions. 
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
 2017-09-04 08:26:57 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. 
Suggested by Victor Gdalevich.
 2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.` 
Contributed by Cynthia Shang.
 2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files. 
Reported by Jesper St John, Aleksandr Rogozin.
 2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process. 
Reported by William Cox.
 2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test. 
Contributed by Cynthia Shang.
 2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. 
Fixed by Cynthia Shang.
 2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. 
Contributed by Cynthia Shang.
 2017-08-01 10:34:53 -04:00
David Steele
fa992433da Update CI code to prevent deprecated VM from being rewritten. 2017-07-27 11:31:42 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups. 
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
 2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command. 
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
 2017-07-21 17:32:41 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. 
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
 2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements: 
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
 2017-06-24 10:59:00 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options. 
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
 2017-06-22 18:22:49 -04:00
David Steele
6327799c9a Use old trusty image until new image works with RHEL6. 2017-06-22 18:13:59 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00
David Steele
8b9965abcb Remove process-max option. 
Parallelism is now tested in a more targeted manner and the high level option is no longer needed.
 2017-06-21 19:21:16 -04:00
David Steele
2348693389 Reduce process redundancy in mock tests. 2017-06-21 18:47:25 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
73274fc608 Remove Ubuntu 12.04 from CI now that it is EOL (it can still be run manually). 2017-06-21 17:02:01 -04:00
David Steele
c69baa2de1 Install sudo in base containers rather than on demand. 2017-06-21 16:51:41 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
edada21866 Rename archive-push unit test. 
This is part of the archive consolidation but is committed separately so that git will follow the rename.
 2017-06-21 08:45:58 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor. 
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
 2017-06-15 15:32:10 -04:00
David Steele
269ae4556d Fix timeouts in ExecuteTest to speed multi-process testing. 2017-06-15 15:20:02 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor: 
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
 2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process. 
Fixed by Cynthia Shang.
 2017-05-16 11:28:51 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. 
Fixed by Cynthia Shang.
 2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version. 
Fixed by Cynthia Shang.
 2017-05-12 15:49:14 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid. 
Suggested by Yogesh Sharma.
 2017-04-12 09:53:07 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings. 
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
 2017-04-10 17:23:39 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages. 
Contributed by Cynthia Shang.
 2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm. 
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
 2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers. 
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
 2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring. 
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
 2017-04-07 10:31:49 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade. 
Contributed by Cynthia Shang.
 2017-04-06 19:36:37 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash. 
Refactor File->list() and fileList() to accept optional parameters.
 2017-04-04 21:17:19 -04:00
David Steele
9eae36a702 Fixed flapping archive stop tests. 2017-04-04 21:15:22 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command. 
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
 2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. 
Contributed by Cynthia Shang.
 2017-03-22 19:03:28 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. 
Reported by Stephen Frost.
 2017-03-13 15:50:13 -04:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. 
Reported by blogh.
 2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements: 
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
 2017-03-02 11:47:23 -05:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment. 
Reported by Stephen Frost.
 2017-03-01 14:15:46 -05:00
David Steele
8161ab1c50 Temporarily disable flapping keep-alive test. 2017-02-26 12:58:48 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
023578c008 Make package tests work with dev builds. 2017-02-21 14:22:03 -05:00
David Steele
0e9bc513c9 Suppress lint on all but the first test. 2017-02-21 12:54:27 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest(). 
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
 2017-02-21 11:56:55 -05:00