David Steele
f7123c1bbf
v2.03: Single Executable to Deploy
...
Bug Fixes:
* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)
Improvements:
* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52
Embed exported C functions and Perl modules directly into the pgBackRest executable.
2018-05-22 12:53:08 -04:00
David Steele
52bc073234
Add stack trace macros to all functions.
...
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c
Test harness improvements.
...
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
790f7c7312
Enforce full branch coverage in C code.
2018-05-05 11:50:11 -04:00
David Steele
88a633da17
Divide tests into three types (unit, integration, performance).
...
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.
Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 09:12:25 -04:00
David Steele
dce11ce3cf
Generate code counts for all source files.
...
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00
David Steele
3e22977484
Use pip 9.03 in test VMs.
...
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
2018-04-16 15:25:50 -04:00
David Steele
f0250dab4b
Move async forking and more error handling to C.
...
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
741acfd779
Move test definitions to test/define.yaml.
...
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 10:19:24 -04:00
David Steele
b77b425096
Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave.
2018-04-07 19:44:00 -04:00
David Steele
82751b3b51
Split debug and assert code into separate headers.
...
Assert can be used earlier because it only depends on the error-handler and not logging.
2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad
More C type cleanup.
...
* Remove typec.h entirely.
* Order all typdefs above local includes.
2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3
Allow MemContext objects to be copied to a new parent.
...
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4
Cleanup C types.
...
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
2018-04-05 22:38:11 -04:00
David Steele
bd25248df0
Abstract Posix storage driver code into a separate module.
2018-04-05 14:48:16 -04:00
David Steele
93fdb98d15
Storage object improvements.
...
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
David Steele
635caff573
Add ASSERT() that is preserved in production builds.
2018-03-30 19:10:34 -04:00
David Steele
c1ab7b3c98
Start work on C handle io object and use it to output help.
2018-03-25 11:39:24 -04:00
David Steele
dabf727d91
Include VM type in gcov path to avoid conflicts between VMs with different architectures.
2018-03-21 16:08:08 -04:00
David Steele
07f38f584a
Use lcov for C unit test coverage reporting.
...
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.
Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
2018-03-19 23:33:28 -04:00
David Steele
0c313713b1
Use clang for static code analysis during lint testing.
...
Nothing found except for some functions that should have been marked __noreturn__.
2018-03-18 13:32:19 -04:00
David Steele
f0451c1494
Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings.
2018-03-15 13:53:03 -04:00
David Steele
4b6cd92a4c
Improve debugging.
...
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a
Add ASSERT_DEBUG() macro for debugging.
...
Replace all current assert() calls except in tests that can't use the debug code.
2018-03-12 14:31:22 -04:00
David Steele
03f1082e86
Improve logging.
...
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
94494a632f
Update pip before installing awscli.
2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd
Improve code generation performance.
2018-03-09 23:00:20 -05:00
David Steele
e331564c90
Add new test for Common::Io::Process.
...
Show that output on stderr will raise an exception on close() even if the exit code is 0.
2018-03-08 12:36:55 -05:00
Cynthia Shang
45ffe37e4f
Add cfgOptionTest().
...
Update cfgOption() calls that are better implemented as cfgOptionTest().
Contributed by Cynthia Shang.
2018-03-05 18:51:48 -05:00
David Steele
406acf9cba
Enable -Wconversion and silence new warnings.
2018-03-02 12:07:12 -05:00
David Steele
5b491b3035
Add CentOS/RHEL package builds.
2018-02-26 19:32:27 -05:00
David Steele
9070325760
Improve performance of HTTPS client.
...
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
2018-02-18 14:53:27 -05:00
David Steele
72cbb9a9d4
Call Perl with built-in interpreter instead of execvp().
...
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
5f2884cb29
Suppress coverage failures for Archive/Push/Async on Travis.
...
The coverage report shows some code as never being run -- but that makes no sense because the tests pass. This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.
Suppress for now with a plan to implement LCOV for the C unit tests.
2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d
Build C binary/library for C unit tests that require Perl.
2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48
Move Perl option JSON generation to a new module.
...
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5
Generate suppressions when running Valgrind.
...
This makes it easier to suppress errors that are only occurring on Travis.
2018-02-14 09:18:27 -05:00
David Steele
305a3e1761
Add int64 variant type.
2018-02-09 13:52:02 -05:00
David Steele
1659598cfe
Create a master list of errors in build/error.yaml.
...
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5
Update URL for Debian package repository.
2018-02-08 14:49:35 -05:00
David Steele
c3f47bf240
Move config parsing out of Perl tests.
...
cfgParseTest() is provided in the C library for parsing configs in unit tests.
2018-02-05 12:32:30 -05:00
David Steele
be90028100
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
...
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
David Steele
a91a648019
Config parsing no longer implemented in Perl.
...
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05
Improve speed of C unit tests.
...
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
2018-01-26 16:41:17 -05:00
David Steele
b58a45e067
Implement help command in C.
2018-01-23 13:34:24 -05:00
David Steele
404ec30d03
Move Perl help test to real module.
2018-01-23 12:58:35 -05:00
David Steele
2afb73da1c
Improve String, StringList, and List objects.
...
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b
The archive-push command is now partially coded in C.
...
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
David Steele
7c73f4f6c9
Add command and exit modules.
2018-01-17 09:15:51 -05:00
David Steele
39cb971afb
Add log module.
2018-01-16 13:52:20 -05:00
David Steele
eb452c8add
Improvements to Storage object.
...
* storagePath() and storageList() commands
* Helper functions for local and spool storag
2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f
Improve error module.
...
Add functions to convert error codes to C errors and handle system errors.
2018-01-16 13:29:27 -05:00
David Steele
d633417e5d
Add Wait object and time module.
2018-01-16 13:05:00 -05:00
David Steele
721d1127a0
Add RegExp object.
2018-01-16 12:57:23 -05:00
David Steele
83ddd75d82
Improvements to String object.
...
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
2018-01-16 12:47:08 -05:00
David Steele
d992851456
Rename Perl tests so they don't conflict with their C counterparts.
2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50
Add stanza-delete command to cleanup unused stanzas.
...
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd
Config parsing implemented in C.
2017-12-22 23:27:49 -05:00
David Steele
d8c575801d
Add Ini object.
2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff
Add Storage object.
2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43
Add KeyValue object.
2017-12-22 22:05:37 -05:00
David Steele
2add6cef95
Add Variant and VariantList objects.
2017-12-22 21:50:27 -05:00
David Steele
c4b30af645
Add Buffer object.
2017-12-22 19:56:01 -05:00
David Steele
f89b027dec
Improve String and StringList modules.
2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e
Run valgrind on all C unit tests.
...
Also refactor code to make valgrind happy.
2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c
No longer run master branch through CI.
...
The integration branch will be run through CI and then pushed to master with github status checks.
2017-12-19 17:03:17 -05:00
David Steele
8d4674031e
More config parsing in C in preparation for all config parsing in C.
2017-12-05 10:09:07 -05:00
David Steele
915ae5662a
Implement version command in C.
2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb
Add unit tests for the Manifest module.
...
Also minor changes to Manifest module, mostly for test reproducibility.
Contributed by Cynthia Shang.
2017-11-28 11:44:24 -05:00
David Steele
74d6398ad2
The pgbackrest executable is now a C binary instead of Perl.
...
This allows certain time-critical commands (like async archive-push) to run more quickly.
2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18
Add List and StringList objects to simplify list handling.
2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b
Add String object to simplify string handling.
2017-11-26 17:49:10 -05:00
David Steele
7cf955425e
The C library is now required.
...
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
c77fc1fa61
Refactor protocol param generation into a new function.
...
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
David Steele
8ccfaa9ad3
Designate a single distro (Ubuntu 16.04) for C coverage testing
...
Enable optimizations on the distros that don't do coverage testing.
2017-11-20 09:05:55 -05:00
David Steele
3b43047e0d
Disable -Wclobber compiler warning.
...
...because it is mostly useless but keep the rest of of -Wextra.
2017-11-19 16:44:33 -05:00
David Steele
500d6b4b66
Automated tests for 32-bit i386/i686 architecture.
2017-11-18 20:14:14 -05:00
David Steele
b701cd5a5f
Designate a single distro (Ubuntu 16.04) for Perl coverage testing.
...
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97
Fixed an issue retrieving WAL for old database versions.
...
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.
Also clean up some issues when the same db version/id appears multiple times in the history.
Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00
David Steele
22fe684b1f
Only check expect logs on CentOS 7.
...
Variations in distros cause false negatives in tests but don't add much value.
2017-11-14 17:16:39 -05:00
David Steele
b8746f368d
Inflate performance improvement for gzip filter and full unit test coverage.
2017-11-14 15:12:31 -05:00
David Steele
bcdfc7d0b5
Enable additional warnings for C builds.
2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a
Repository encryption support.
...
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
David Steele
8d6a08a32b
Library code for repository encryption support.
2017-11-03 13:57:58 -04:00
David Steele
ac542788da
Convert configuration definitions from auto-generated functions to auto-generated data structures.
2017-11-02 08:14:13 -04:00
David Steele
f57e376c44
Update Debian/Ubuntu containers to download lastest version of pip.
2017-10-30 09:29:46 -04:00
David Steele
d989cf8ac2
Replace dynamically built class hierarchies in I/O layer with fixed parent() calls.
2017-10-22 19:07:17 -04:00
David Steele
3553ccae44
Add I/O performance tests.
2017-10-22 13:27:47 -04:00
David Steele
f4524aeaa9
Add base64 encode/decode.
2017-10-18 11:14:26 -04:00
David Steele
b1efd598cf
Add PostgreSQL versions to Debian VMs for testing.
2017-10-16 18:21:47 -04:00
David Steele
583a76f605
Add C memory contexts.
2017-10-16 11:25:49 -04:00
David Steele
6f5186f9e6
Convert config tests into C unit tests.
2017-10-16 11:13:51 -04:00
David Steele
9d78948a14
Improvements to page checksum module and tests:
...
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
2017-10-16 11:03:06 -04:00
David Steele
904b83747c
Perl error handler recognizes errors thrown from the C library.
2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa
Add C error handler.
...
Basic try ... catch ... finally pattern to simplify error handling in C.
2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5
Add C unit test infrastructure.
2017-10-12 12:55:48 -04:00
David Steele
abea4d1fd5
Remove Debian test repo after PostgreSQL 10 release.
2017-10-05 12:33:13 -04:00
David Steele
c857015a38
Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization.
2017-09-19 10:03:53 -04:00
David Steele
7c95a2f4de
Updates to installation documentation.
...
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
2017-09-08 21:50:00 -07:00