David Steele
c1ab7b3c98
Start work on C handle io object and use it to output help.
2018-03-25 11:39:24 -04:00
David Steele
9001b9b957
Replace THROW_ON_SYS_ERROR() with THROW_SYS_ERROR().
...
The former macro was hiding missing branch coverage for critical error handling.
2018-03-24 14:11:29 -04:00
David Steele
dabf727d91
Include VM type in gcov path to avoid conflicts between VMs with different architectures.
2018-03-21 16:08:08 -04:00
David Steele
c6457b070f
Full branch coverage for config/parse module.
2018-03-21 13:46:08 -04:00
David Steele
b234f43c9d
Refactor usec to msec in common/time.c.
...
The implementation provides usec resolution but this is not needed in practice and it makes the interface more complicated due to the extra zeros.
2018-03-21 09:18:48 -04:00
David Steele
31830bdc55
Full branch coverage for all common/type modules.
2018-03-20 16:08:52 -04:00
David Steele
a2030da200
Specify prefix for lcov output.
...
This prevents pages from moving around depending on the subtest.
2018-03-20 16:01:38 -04:00
David Steele
f4ed36d128
Add switch() to lcov branch exclusions.
...
lcov requires default to show complete coverage but --Wswitch-enum enforces all enum values be present so no default is needed.
Add documentation for each branch exclusion.
2018-03-20 10:17:53 -04:00
David Steele
d34142dee8
Update lcov settings.
...
Mark any coverage less that 100% as yellow, less that 90% as red.
Add comments to config file and make genhtml use it as well as lcov.
2018-03-20 09:47:51 -04:00
David Steele
07f38f584a
Use lcov for C unit test coverage reporting.
...
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.
Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
2018-03-19 23:33:28 -04:00
David Steele
45c6a0e864
Remove hard-coded release version from loadTest.c.
2018-03-19 21:03:42 -04:00
David Steele
5b0748cf93
Always autogenerate code if --smart is not specified.
2018-03-19 20:25:55 -04:00
Cynthia Shang
f0a4264f77
Fix manifest test in the check command when tablespaces are present.
...
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
2018-03-19 19:54:03 -04:00
David Steele
8e9e97728c
Remove --smart from --expect tests.
...
This ensures that new binaries are built before running the tests.
2018-03-19 17:38:44 -04:00
David Steele
2c1687721e
Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected.
...
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.
Reported by Clinton Adams.
2018-03-19 13:08:42 -04:00
David Steele
0c313713b1
Use clang for static code analysis during lint testing.
...
Nothing found except for some functions that should have been marked __noreturn__.
2018-03-18 13:32:19 -04:00
David Steele
f0451c1494
Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings.
2018-03-15 13:53:03 -04:00
David Steele
8a1ce42c30
Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line.
...
Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem.
Update command documentation to omit secure options that cannot be specified on the command-line.
Reported by Brad Nicholson.
2018-03-15 12:02:09 -04:00
David Steele
dd3ce70810
Config test code writes secure options to a file instead of passing on the command-line.
2018-03-15 11:09:46 -04:00
David Steele
4fb75c9cc1
Improve conversion of C exceptions to Exception objects.
...
Colons in the message would prevent all of the message from being loaded into the Exception object.
2018-03-15 11:03:28 -04:00
David Steele
4b6cd92a4c
Improve debugging.
...
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a
Add ASSERT_DEBUG() macro for debugging.
...
Replace all current assert() calls except in tests that can't use the debug code.
2018-03-12 14:31:22 -04:00
Cynthia Shang
dd31ae832d
Port most of Config::Config::configLoad() from Perl to C.
...
Contributed by Cynthia Shang.
2018-03-12 11:25:20 -04:00
David Steele
03f1082e86
Improve logging.
...
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
94494a632f
Update pip before installing awscli.
2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd
Improve code generation performance.
2018-03-09 23:00:20 -05:00
David Steele
b987f05f35
Improve bin and libc build performance.
2018-03-09 21:24:30 -05:00
David Steele
c32ee34612
Build with -DNDEBUG by default but disable for testing.
2018-03-09 18:28:34 -05:00
David Steele
fbe4c40386
Move logic for setting Perl configuration defaults to C.
2018-03-09 09:30:50 -05:00
David Steele
919635b6f4
Set config before Main::main() call to avoid secrets being exposed in a stack trace.
2018-03-08 16:24:16 -05:00
Cynthia Shang
34898ce471
Error when multiple arguments are set in the config file for an option that does not accept multiple arguments.
...
Contributed by Cynthia Shang.
2018-03-08 14:15:05 -05:00
David Steele
e331564c90
Add new test for Common::Io::Process.
...
Show that output on stderr will raise an exception on close() even if the exit code is 0.
2018-03-08 12:36:55 -05:00
David Steele
62a9bdbd4c
Fix issue passing --no-config to embedded Perl.
...
Reported by Ibrahim Edib Kokdemir.
2018-03-08 10:43:09 -05:00
Cynthia Shang
45ffe37e4f
Add cfgOptionTest().
...
Update cfgOption() calls that are better implemented as cfgOptionTest().
Contributed by Cynthia Shang.
2018-03-05 18:51:48 -05:00
David Steele
c07aa2ae19
Improve code documentation in config module.
2018-03-02 22:42:02 -05:00
David Steele
406acf9cba
Enable -Wconversion and silence new warnings.
2018-03-02 12:07:12 -05:00
David Steele
af805af044
Remove Debian package patch now that it has been merged upstream.
2018-03-01 16:59:13 -05:00
David Steele
5d1722b8ef
Disable console display of coverage for C files.
...
Devel::Cover does not handle it well.
2018-03-01 13:32:51 -05:00
David Steele
57d70a5649
Remove RHEL package patch since upstream has been updated.
2018-03-01 13:31:03 -05:00
David Steele
5b491b3035
Add CentOS/RHEL package builds.
2018-02-26 19:32:27 -05:00
David Steele
64a7196cb6
Rename retention-* options to repo-retention-*.
2018-02-19 15:07:24 -05:00
Cynthia Shang
f75ba7db94
Update backup host to repository host in the documentation.
...
Contributed by Cynthia Shang.
2018-02-19 13:17:58 -05:00
Cynthia Shang
4352407777
Option rename cleanup based on review.
...
* Remove --reset tags from v1 options.
* Use constants for repo prefix.
* Specify CFGDEF_INDEX_PG in option structure rather than adding in code.
* Fix error message references to "backup host".
Contributed by Cynthia Shang.
2018-02-19 10:12:51 -05:00
David Steele
9070325760
Improve performance of HTTPS client.
...
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
2018-02-18 14:53:27 -05:00
David Steele
27678f6188
Clean up and document Makefile.
2018-02-18 10:44:58 -05:00
David Steele
72cbb9a9d4
Call Perl with built-in interpreter instead of execvp().
...
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
81952c41f4
Report coverage errors via the console.
...
This helps with debugging coverage issues on remote services like Travis.
2018-02-14 17:13:18 -05:00
David Steele
5f2884cb29
Suppress coverage failures for Archive/Push/Async on Travis.
...
The coverage report shows some code as never being run -- but that makes no sense because the tests pass. This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.
Suppress for now with a plan to implement LCOV for the C unit tests.
2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d
Build C binary/library for C unit tests that require Perl.
2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48
Move Perl option JSON generation to a new module.
...
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5
Generate suppressions when running Valgrind.
...
This makes it easier to suppress errors that are only occurring on Travis.
2018-02-14 09:18:27 -05:00
Cynthia Shang
00e9aca872
Fixed issue where --reset-* was not passing the default value to Perl.
...
Contributed by Cynthia Shang.
2018-02-13 15:58:14 -05:00
David Steele
7c58fe7451
Use 64-bit integers for config options.
...
In particular, archive-queue-max requires a greater range than is provided by a 32-bit integer.
2018-02-09 13:54:33 -05:00
David Steele
305a3e1761
Add int64 variant type.
2018-02-09 13:52:02 -05:00
David Steele
1659598cfe
Create a master list of errors in build/error.yaml.
...
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5
Update URL for Debian package repository.
2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b
Rename -no to -reset for non-boolean options.
2018-02-06 11:26:06 -05:00
David Steele
808b6000eb
Fix some bugs in C config code:
...
1) Error when the same option is defined multiple times in a section using alternate names.
2) Fix logging of invalid command error.
3) Warn when a stanza-only option is in a global section.
Also, make a note to add validation of section names to the check command.
Per review by Cynthia Shang.
2018-02-05 15:48:26 -05:00
David Steele
d35757c05f
Fix missing log-level-stderr option for stanza-* commands.
2018-02-05 15:19:28 -05:00
David Steele
c3f47bf240
Move config parsing out of Perl tests.
...
cfgParseTest() is provided in the C library for parsing configs in unit tests.
2018-02-05 12:32:30 -05:00
David Steele
db21b7a360
Cleanup usage of internal options.
...
Apply internal to options that need to be read to determine locality but should not appear in the help.
2018-02-05 11:50:09 -05:00
David Steele
be90028100
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
...
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e
Improve stanza-create command so that it does not error when the stanza already exists.
...
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
David Steele
298b33988c
Fix non-compliant formatting for function declarations.
...
Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
Cynthia Shang
bacc08792d
Ensure latest db-id is selected on when matching archive.info to backup.info.
...
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).
Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0
Fixed inability to restore a single database contained in a tablespace using --db-include.
...
Fixed by Cynthia Shang.
2018-01-30 16:13:54 -05:00
David Steele
a91a648019
Config parsing no longer implemented in Perl.
...
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05
Improve speed of C unit tests.
...
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
2018-01-26 16:41:17 -05:00
David Steele
b58a45e067
Implement help command in C.
2018-01-23 13:34:24 -05:00
David Steele
404ec30d03
Move Perl help test to real module.
2018-01-23 12:58:35 -05:00
David Steele
3dd882a79e
Add internal flag to config definition.
2018-01-23 12:49:04 -05:00
David Steele
41bd5e8148
Improve KeyValue object.
...
Add kvGetList() to get a value as a list (of 1) even if it is a scalar.
2018-01-23 12:40:02 -05:00
David Steele
2afb73da1c
Improve String, StringList, and List objects.
...
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b
The archive-push command is now partially coded in C.
...
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
David Steele
1f39a34abc
Move lock release later in exitSafe().
...
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
2018-01-17 10:31:21 -05:00
David Steele
7c73f4f6c9
Add command and exit modules.
2018-01-17 09:15:51 -05:00
David Steele
39cb971afb
Add log module.
2018-01-16 13:52:20 -05:00
David Steele
eb452c8add
Improvements to Storage object.
...
* storagePath() and storageList() commands
* Helper functions for local and spool storag
2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f
Improve error module.
...
Add functions to convert error codes to C errors and handle system errors.
2018-01-16 13:29:27 -05:00
David Steele
bffc6c49b3
Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work.
2018-01-16 13:23:08 -05:00
David Steele
d633417e5d
Add Wait object and time module.
2018-01-16 13:05:00 -05:00
David Steele
721d1127a0
Add RegExp object.
2018-01-16 12:57:23 -05:00
David Steele
05c731692b
Add varStrForce() to Variant object.
2018-01-16 12:54:06 -05:00
David Steele
83ddd75d82
Improvements to String object.
...
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
2018-01-16 12:47:08 -05:00
David Steele
d992851456
Rename Perl tests so they don't conflict with their C counterparts.
2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50
Add stanza-delete command to cleanup unused stanzas.
...
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd
Config parsing implemented in C.
2017-12-22 23:27:49 -05:00
David Steele
d8c575801d
Add Ini object.
2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff
Add Storage object.
2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43
Add KeyValue object.
2017-12-22 22:05:37 -05:00
David Steele
2add6cef95
Add Variant and VariantList objects.
2017-12-22 21:50:27 -05:00
David Steele
c4b30af645
Add Buffer object.
2017-12-22 19:56:01 -05:00
David Steele
f89b027dec
Improve String and StringList modules.
2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e
Run valgrind on all C unit tests.
...
Also refactor code to make valgrind happy.
2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c
No longer run master branch through CI.
...
The integration branch will be run through CI and then pushed to master with github status checks.
2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f
Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster.
...
Fixed by Cynthia Shang.
Reported by Stephen Frost.
2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33
Fixed an issue where WAL was not expired on PostgreSQL 10.
...
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.
Reported by Adam Brusselback.
2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82
Improve check command to verify that the backup manifest can be built.
...
Contributed by Cynthia Shang.
2017-12-13 11:16:27 -05:00
David Steele
99a0721631
Allow any non-boolean, non-command-line option to be negated.
...
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.
Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340
Fixed an issue that suppressed locality errors for backup and restore.
...
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.
Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00