1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

705 Commits

Author SHA1 Message Date
David Steele
9ba95e993b Use retries to wait for test S3 server to start.
The prior method of tailing the docker log no longer seems reliable.  Instead, keep retrying the make bucket command until it works and show the error if it times out.
2019-06-13 17:58:33 -04:00
David Steele
6ff3325c77 Enforce requiring repo-cipher-pass at config parse time.
This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.

Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
2019-06-05 11:43:17 -04:00
David Steele
92e04ea9f4 Remove per-stanza repo cache clear during testing.
This was not being used and is not supported by the equivalent C code.
2019-06-04 10:34:19 -04:00
David Steele
12bca3c43e Add CPPFLAGS to compile rules.
This should silence the last of the Debian package warnings.
2019-06-01 09:28:31 -04:00
David Steele
a2ec1253e9 Add code classification exclusion missed in 3e1b06ac. 2019-05-30 10:44:35 -04:00
David Steele
3e1b06acaa Use minio as local S3 emulator in documentation.
The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.

Also, install certificates and CA so that TLS verification can be enabled.
2019-05-27 07:37:20 -04:00
David Steele
86482c7db9 Reduce log level for all expect tests to detail.
The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.

Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-22 18:23:44 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
Cynthia Shang
19d8358cba Update mock/expire module test matrix so expect tests output.
Also add an error message to prevent regression.

Contributed by Cynthia Shang.
2019-05-16 09:53:55 -04:00
Cynthia Shang
18d4cb5741 Bypass database checks when stanza-delete issued with force.
Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted.

Contributed by Cynthia Shang.
Suggested by Roman.
2019-05-15 13:14:58 -04:00
David Steele
5c1d4bcd0d Automate coverage summary report generation.
This report replaces the lcov report that was generated manually for each release.

The lcov report was overly verbose just to say that we have virtually 100% coverage.
2019-05-15 13:04:56 -04:00
David Steele
5bba72b874 Remove -Wswitch-enum compiler option.
The -Wswitch option included in -Wall provides the same level of coverage and allows enum options to be grouped into default.
2019-05-15 12:55:08 -04:00
David Steele
87f36e814e Improve macros and coverage rules that were hiding missing coverage.
The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement.  Improve matching so that all characters in the name must be upper-case for a match.

Some macros with internal branches accepted parameters that might contain conditionals.  This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases.  Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing.  In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing.  In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing.

Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter.

Finally, update tests and code where missing coverage was exposed by these changes.  Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
David Steele
cb00030ee3 Remove dead code missed in 1b486847.
This commit removed all Perl references to spool storage but some stuff was left behind.
2019-05-08 18:58:07 -04:00
David Steele
8c712d89eb Improve type safety of interfaces and drivers.
The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations.  Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been.

Use void pointers instead so that no casts are required.  Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer.

Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type.  This cuts down on accessor methods and the vast majority of those functions were not being used.  Move functions that are still required to .intern.h.

Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
David Steele
28359eea83 Update code count rules missed in 027c2638. 2019-05-02 16:33:23 -04:00
David Steele
027c263871 Add configure script for improved multi-platform support.
Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile.

Each C file must include "build.auto.conf" before all other includes and defines.  This is enforced by test.pl for includes, but it won't detect incorrect define ordering.

Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
David Steele
3505559a80 Update test containers with PostgreSQL minor releases and liblz4.
Update RHEL repos that have changed upstream.  Remove PostgreSQL 9.3 since the RHEL6/7 packages have disappeared.

Remove PostgreSQL versions from U12 that are still getting minor updates so the container does not need to be rebuilt.

LZ4 is included for future development, but this seems like a good time to add it to the containers.
2019-04-24 13:23:32 -04:00
David Steele
1ae8a6a716 Add build-max option to set max build processes.
Currently this controls make processes via -j.
2019-04-23 20:52:03 -04:00
David Steele
c11c936366 Reduce ScalityS3 processes since only two are needed. 2019-04-23 20:19:31 -04:00
David Steele
41f3874822 v2.13: Bug Fixes
Bug Fixes:

* Fix zero-length reads causing problems for IO filters that did not expect them. (Reported by brunre01, jwpit, Tomasz Kontusz, guruguruguru.)
* Fix reliability of error reporting from local/remote processes.
* Fix Posix/CIFS error messages reporting the wrong filename on write/sync/close.
2019-04-18 21:26:02 -04:00
David Steele
3aa521fed0 Fix compile flag accidentally removed in 5ee8388f. 2019-04-10 13:37:24 -04:00
David Steele
1b48684713 The archive-push command is implemented entirely in C.
This new implementation should behave exactly like the old Perl code with the exception of updated log messages.

Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
David Steele
5ee8388f48 Build test harness with the same warnings as code being tested.
The test harness was not being built with warnings which caused some wackiness with an improperly structured switch.  Just use the same warnings as the code being tested.

Also enable warnings on code that is not directly being tested since other code modules are frequently modified during testing.
2019-03-26 08:20:55 +02:00
David Steele
e26d510d0c Use restore command for remote performances tests.
Since archive-push is being moved to C, the Perl remote will no longer work with that command.

Eventually this module will need to be rewritten in C, but for now just use the restore command which is planned to be migrated last.
2019-03-17 22:11:35 +04:00
David Steele
9382283586 Fix issues when a path option is / terminated.
This condition was not being properly checked for in the C code and it caused problems in the info command, at the very least.

Instead of applying a local fix, introduce a new path option type that will rigorously check the format of any incoming paths.

Reported by Marc Cousin.
2019-03-14 13:48:33 +04:00
David Steele
b8ebea6b1c Add separate archive-push-async command.
This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-03-14 13:38:55 +04:00
blogh
e4e2606fce Add additional options to backup.manifest for debugging purposes.
Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging.

It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient.

Contributed by blogh.
2019-03-10 11:03:52 +02:00
David Steele
21f56f64eb Add hints when unable to find a WAL segment in the archive.
When this error happens in the context of a backup it can be a bit mystifying as to why the backup is failing.  Add some hints to get the user started.

These hints will appear any time a WAL segment can't be found, which makes the hint about the check command redundant when the user is actually running the check command, but it doesn't seem worth trying to exclude the hint in that case.

Suggested by Hans-Jürgen Schönig.
2019-03-10 10:38:12 +02:00
David Steele
d441061168 Create test matrix for mock/all to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
David Steele
f7d1d4400f Create test matrix for mock/expire to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests.
2019-03-01 19:04:26 +02:00
David Steele
91622942c2 Create test matrix for mock/archive-stop to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-01 17:12:41 +02:00
Marc Cousin
cb3b4fa24b Enable socket keep-alive on older Perl versions.
The prior method depended on IO:Socket:SSL to push the keep-alive options down to the socket but it only worked for recent versions of the module.

Instead, create the socket directly using IO::Socket::IP if available or IO:Socket:INET as a fallback.  The keep-alive option is set directly on the socket before it is passed to IO:Socket:SSL.

Contributed by Marc Cousin.
2019-02-28 14:33:29 +02:00
David Steele
db4b447be8 The archive-get command is implemented entirely in C.
This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages.

Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
David Steele
9367cc461c Migrate local command to C.
The C local is only used for C commands in the main process.

Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
David Steele
18b62a4220 Only run test-level stack trace by default for unit-tested modules.
This amends 70c30dfb which disabled test tracing in general.

Instead, only enable test tracing by default for modules that are being unit tested. This saves lots of time but still ensures that test tracing is working and helps with debugging in unit tests.

Also rename the option to --debug-test-trace for a clarity.
2019-02-27 17:09:19 +02:00
David Steele
3a05359087 Create test matrix for mock/stanza to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
David Steele
6d3e18b181 Reduce expect log level in mock/stanza tests.
The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior.

Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit.

Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
David Steele
2f081f3ec7 Rename test modules for consistency.
The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
David Steele
d489eb87f7 Create test matrix for mock/archive to increase coverage and reduce tests.
The same test configurations are run on all four test VMs, which seems a real waste of resources.

Vary the tests per VM to increase coverage while reducing the total number of tests.  Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-23 15:59:39 +02:00
David Steele
4a7588e604 Create aliases for test VMs ordered by age.
This will allow for smarter allocation of tests in the next commit.
2019-02-23 15:13:23 +02:00
David Steele
59d7958914 Reduce expect log level in mock/archive tests.
The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior.

Now the archive code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit.

Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-23 15:05:06 +02:00
David Steele
70c30dfb61 Disable test-level stack trace by default.
Detailed stack traces for low-level functions (e.g. strCat, bufMove) can be very useful for debugging but leaving them on for all tests has become quite burdensome in terms of time.  Complex operations like generating JSON on a large KevValue can lead to timeouts even with generous values.

Add a new param, --debug-trace, to enable test-level stack trace, but leave it off by default.
2019-02-22 11:40:30 +02:00
David Steele
d211c2b8b5 Fix possible truncated WAL segments when an error occurs mid-write.
The file write object destructors called close() and finalized the file even if it was not completely written.  This was an issue in both the C and Perl code.

Rewrite the destructors to simply free resources (like file handles) rather than calling the close() method.  This leaves the temp file in place for filesystems that use temp files.

Add unit tests to prevent regression.

Reported by blogh.
2019-02-15 11:52:39 +02:00
David Steele
057e2e2782 Add unimplemented S3 driver method required for archive-get.
This was not being caught because the integration tests for S3 were running remotely and going through the Perl code rather than the new C code.

Implement the exists method for the S3 driver and add tests to prevent a regression.

Reported by mibiio.
2019-02-09 18:57:30 +02:00
David Steele
aa3e5b8c72 Allow primary gid for the test user to be different from uid.
Apparently up until now they have always been the same, which is pretty typical.  However, if they were not then ContainerTest.pm was not happy.
2019-01-30 17:03:17 +02:00
David Steele
8f6d324b2c Fix issue with multiple async status files causing a hard error.
Multiple status files were being created by asynchronous archiving if a high-level error occurred after one or more WAL segments had already been transferred successfully.  Error files were being written for every file in the queue regardless of whether it had already succeeded.  To fix this, add an option to skip writing error files when an ok file already exists.

There are other situations where both files might exist (various fsync and filesystem error scenarios) so it seems best to retry in the case that multiple status files are found rather than throwing a hard error (which then means that archiving is completely stuck).  In the case of multiple status files, a warning will be logged to alert the user that something unusual is happening and the command will be retried.

Reported by fpa-postgres, Joe Ayers, Douglas J Hunley.
2019-01-26 16:59:54 +02:00
David Steele
d245f8eb42 The info command is implemented entirely in C.
The C info code has already been committed but this commit wires it into main.

Also remove the info Perl code and tests since they are no longer called.
2019-01-21 13:51:45 +02:00
David Steele
db24ff8df4 v2.08: Minor Improvements and Bug Fixes
Bug Fixes:

* Remove request for S3 object info directly after putting it. (Reported by Matt Kunkel.)
* Correct archive-get-queue-max to be size type. (Reported by Ronan Dunklau.)
* Add error message when current user uid/gid does not map to a name. (Reported by Camilo Aguilar.)
* Error when --target-action=shutdown specified for PostgreSQL < 9.5.

Improvements:

* Set TCP keepalives on S3 connections. (Suggested by Ronan Dunklau.)
* Reorder info command text output so most recent backup is output last. (Contributed by Cynthia Shang. Suggested by Ryan Lambert.)
* Change file ownership only when required.
* Redact authentication header when throwing S3 errors. (Suggested by Brad Nicholson.)
2019-01-02 22:04:47 +02:00
Cynthia Shang
35bbb5bd68 Reorder info command text output so most recent backup is output last.
After a stanza-upgrade backups for the old cluster are displayed until they expire.  Cluster info was output newest to oldest which meant after an upgrade the most recent backup would no longer be output last.

Update the text output ordering so the most recent backup is always output last.

Contributed by Cynthia Shang.
Suggested by Ryan Lambert.
2018-12-14 18:25:31 -05:00
Cynthia Shang
cbf514e191 Improve info error messages introduced in 74b72df9.
- Add detail to errors when info files are loaded with incorrect encryption settings.
- Throw FileMissingError rather than FileOpenError when both copies of the info file are missing.
- If one file is present (but errors) and the other is missing, then return the error for the file that was present.

Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
David Steele
e73416e9e3 Change file ownership only when required.
Previously chown() would be called even when no ownership changes were required.

In most cases changes are not required and it seems better to perform an extra stat() rather than an extra chown().

Also add unit tests for owner() since there weren't any.
2018-12-05 17:56:47 -05:00
David Steele
cc6447356e Fix test binary name for gprof.
This got missed in 1f8931f7 when the test binary was renamed.

Also output call graph along with the flat report.  The flat report is generally most useful but it doesn't hurt to have both.
2018-12-05 09:15:45 -05:00
David Steele
74b72df9db Improve error message when info files are missing/corrupt.
The previous error message only showed the last error.  In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked.

Show both errors below a generic "unable to load" error.  Details are now given explaining exactly why the primary and copy failed.

Previously if one file could not be loaded a warning would be output.  This has been removed because it is not clear what the user should do in this case.  Should they do a stanza-create --force?  Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
David Steele
7c2fcb63e4 Enable encryption for archive-get command in C.
The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.

The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
315aa2c451 Conditional compilation of Perl logic in exit.c.
This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.
2018-11-25 08:39:41 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
beae375330 Enable S3 storage for archive-get command in C.
The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.

The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.
2018-11-23 12:18:07 -05:00
David Steele
ac426bc456 New test containers with static test certificates.
Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.

The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.

Two new certificates have been defined. See test/certificate/README.md for details.

The old dynamic certificates will be retained until they are replaced.
2018-11-21 18:13:37 -05:00
David Steele
bc25db5667 Add interface objects for libxml2.
Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.

This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-20 20:40:11 -05:00
David Steele
f743d4e924 Add testRepoPath() to let C unit tests know where the code repository is located.
This allows a C unit test to access data in the code repository that might be useful for testing.

Add testRepoPathSet() to set the repository path.

In passing remove extra whitespace in the TEST_RESULT_VOID() macro.
2018-11-20 15:48:56 -05:00
David Steele
8c7e97a369 Clarify comment about main.c being excluded from unit testing.
Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:08:26 -05:00
David Steele
acb579c469 Tighten limits on code coverage context selection.
If the last } of a function was marked as uncovered then the context selection would overrun into the next function.

Start checking context on the current line to prevent this.  Make the same change for start context even though it doesn't seem to have an issue.
2018-11-13 10:37:58 -05:00
David Steele
7107cc68d2 Expand context shown in coverage and update colors.
Too few lines were shown for coverage context so show the entire function if it has any missing coverage.

Update colors to work with light and dark browser modes.
2018-11-12 18:11:16 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
1f8931f732 Improve single test run performance.
Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes).  This simplifies some of the rsync logic and allows make to do what it does best.

Also split build flag files into test, harness, and build to reduce rebuilds.  Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 16:34:04 -04:00
David Steele
7794ab50dc Preserve contents of C unit test build directory between test.pl executions.
The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.

Save the important build flags in a file so the new execution knows whether the build contents can be reused.
2018-11-02 11:56:13 -04:00
Cynthia Shang
34c63276cd Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected.
There are a number of cases where a checksum delta is more appropriate than the default time-based delta:

* Timeline has switched since the prior backup
* File timestamp is older than recorded in the prior backup
* File size changed but timestamp did not
* File timestamp is in the future compared to the start of the backup
* Online option has changed since the prior backup

A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch.  In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention.

Now, a checksum delta will be performed if the backup type is incr/diff.  This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted.

Contributed by Cynthia Shang.
2018-11-01 11:31:25 -04:00
David Steele
cca7a4ffd4 Retry all S3 5xx errors rather than just 500 internal errors.
We were already retrying 500 errors but 503 (rate-limiting) errors were not being retried and would cause an instant failure which aborted the command.

There are only two 5xx errors currently implemented by S3 but instead of adding 503 simply retry all 5xx errors. This is consistent with the http definition of this error class, "the server failed to fulfill an apparently valid request."

Suggested by Craig A. James.
2018-10-30 16:45:42 -04:00
David Steele
286f7e5011 Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
This calculation was missed when the WAL segment size was made dynamic in preparation for PostgreSQL 11.

Fix the calculation by checking the actual WAL file sizes instead of using an estimate based on WAL segment size.  This is more accurate because it takes into account .history and .backup files, which are smaller.  Since the calculation is done in the async process the additional processing time should not adversely affect performance.

Remove the PG_WAL_SIZE constant and instead use local constants where the old value is still required.  This is only the case for some tests and PostgreSQL 8.3 which does not provide a way to get the WAL segment size from pg_control.
2018-10-27 20:00:00 +01:00
David Steele
41b00dc204 Fix issue with archive-push-queue-max not being honored on connection error.
If an error occurred while acquiring a lock on a remote server the error would be reported correctly, but the queue max detection code was not reached.  The tests failed to detect this because they fixed the connection before queue max, allowing the ccde to be reached.

Move the queue max code before the lock so it will run even when remote connections are not working.  This means that no attempt will be made to transfer WAL once queue max has been exceeded, but it makes it much more likely that the code will be reach without error.

Update tests to continue errors up to the point where queue max is exceeded.

Reported by Lardière Sébastien.
2018-10-27 16:57:57 +01:00
David Steele
9ae3d8c46a Install nodejs from deb.nodesource.com.
The standard npm packages on Ubuntu 18.04 suddenly required libssl1.0 which broke the pgbackrest package builds. Installing nodejs from deb.nodesource.com seems to work fine with standard libssl.

This package is required by ScalityS3 which is used for local S3 testing.
2018-10-15 23:13:08 +01:00
David Steele
d038b9a029 Support configurable WAL segment size.
PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB.

There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup.  Both operations require the WAL segment size to properly build a list.

Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database.

The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead.  This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them.  While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL.

Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
b6b2c915b2 Allow hashSize() to run on remote storage.
Apparently we never needed to run this function remotely.

It will be needed by the backup checksum delta feature, so implement it now.

Contributed by Cynthia Shang.
2018-09-18 11:39:48 -04:00
David Steele
e55d733041 Add -ftree-coalesce-vars option to unit test compilation.
This is a workaround for inefficient handling of many setjmps in gcc >= 4.9.  Setjmp is used in all error handling, but in the unit tests each test macro contains an error handling block so they add up pretty quickly for large unit tests.

Enabling -ftree-coalesce-vars in affected versions reduces build time and memory requirements by nearly an order of magnitude.  Even so, compiles are much slower than gcc <= 4.8.

We submitted a bug for this at: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87316
Which was marked as a duplicate of: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63155
2018-09-17 11:38:10 -04:00
David Steele
b5f749b21c Add CIFS driver to storage helper for read-only repositories.
For read-only repositories the Posix and CIFS drivers behave exactly the same.  Since that's all we support in C right now it's valid to treat them as the same thing.  An assertion has been added to remind us to add the CIFS driver before allowing the repository to be writable.

Mostly we want to make sure that the C code does not blow up when the repository type is CIFS.
2018-09-16 18:41:30 -04:00
David Steele
4119ce208d Move test expect log out of the regular test directory.
Storing the expect log (created by common/harnessLog) in the regular test directory was not ideal.  It showed up in tests and made it difficult to clear the test directory between each run.

Move the expect log to a purpose-built directory one level up so it does not interfere with regular testing.
2018-09-16 15:58:46 -04:00
David Steele
f0ed89f21f Allow C or Perl coverage to run on more than one VM.
C or Perl coverage tests can now be run on any VM provided a recent enough version of Devel::Cover or lcov is available.

For now, leave u18 as the only VM to run coverage tests due to some issues with older versions of lcov.
2018-09-15 13:27:06 -04:00
David Steele
31cdd9d20b Remove compiler warnings that are not valid for u16. 2018-09-15 08:23:55 -04:00
David Steele
aeb1fa3dfb Don't perform valgrind when requested.
The --no-valgrind flag was not being honored.  It's not clear if this flag ever worked, but it does now.
2018-09-13 19:12:40 -04:00
Cynthia Shang
e351b8c67c Improve info command to display the stanza cipher type.
Contributed by Cynthia Shang.
Suggested by Douglas J Hunley.
2018-09-10 13:09:45 -04:00
David Steele
c688bc8627 Improve support for special characters in filenames.
% characters caused issues in backup/restore due to filenames being appended directly into a format string.

Reserved XML characters (<>&') caused issues in the S3 driver due to improper escaping.

Add a file with all common special characters to regression testing.
2018-09-10 10:54:34 -04:00
David Steele
f7fc8422f7 Make Valgrind return an error even when a non-fatal issue is detected.
By default Valgrind does not exit with an error code when a non-fatal error is detected, e.g. unfreed memory.  Use the --error-exitcode option to enabled this behavior.

Update some minor issues discovered in the tests as a result.  Luckily, no issues were missed in the core code.
2018-09-07 16:50:01 -07:00
David Steele
de1b74da0c Move encryption in mock/archive tests to remote tests.
The new archive-get C code can't run (yet) when encryption is enabled.  Therefore move the encryption tests so we can test the new C code.  We'll move it back when encryption is enabled in C.

Also, push one WAL segment with compression to test decompression in the C code.
2018-09-06 09:35:34 -07:00
David Steele
6361a06181 Fix incorrectly reported error return in info logging.
A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.

Also improve info messages when an archive segment is or is not found.
2018-09-04 21:46:41 -04:00
David Steele
375ff9f9d2 Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL.
Previously an error would be generated if other files were present and not owned by the PostgreSQL user.  This hasn't been a big deal in practice but it could cause issues.

Also add tests to make sure the same logic applies with links to files, i.e. all other files in the directory should be ignored.  This was actually working correctly, but there were no tests for it before.
2018-08-31 16:06:40 -04:00
David Steele
d41570c37a Improve log file names for remote processes started by locals.
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes.  Not only was the local command used for the naming of log files but the process id was not pass through.  This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.

Instead, pass the real command and process id to the remote.  This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
2018-08-31 11:31:13 -04:00
David Steele
70514061fd Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path.
Relative link paths were being combined with the paths of previous links (relative or absolute) due to the $strPath variable being modified in the current iteration rather than simply being passed to the next level of recursion.

This issue did not affect absolute links and relative tablespace links were caught by other checks, though the error was confusing.

Reported by Cynthia Shang.
2018-08-30 16:27:36 -04:00
David Steele
c638490451 Documentation updates for exclude feature based on review.
Reviewed by Cynthia Shang.
2018-08-28 16:49:29 -04:00
David Steele
14cde54b37 Limit manifest build recursion (i.e. links followed) to sixteen levels to detect link loops. 2018-08-28 16:27:10 -04:00
David Steele
a6cecf7d5e Prevent manifest from being built more than once. 2018-08-28 16:22:30 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
Cynthia Shang
eb30d88b6a Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta.
Contributed by Cynthia Shang.
2018-08-24 16:50:33 -04:00
David Steele
0ed37ab9e7 Update Archive::Info->archiveIdList() to return a valid error code instead of unknown. 2018-08-24 12:13:10 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
8a8738308c Enable -Wvla. 2018-08-22 14:48:37 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
5dc8a2ec08 storageFileRead() accepts a buffer for output rather than creating one.
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
2018-07-17 19:01:54 -04:00
Cynthia Shang
0acf705416 Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
Contributed by Cynthia Shang.
2018-07-16 17:25:15 -04:00
David Steele
22adb395db Add zero-length file to mock/all test. 2018-07-05 15:40:50 -04:00
David Steele
db17973cd0 Fix critical bug in resume that resulted in inconsistent backups.
A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".

Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 14:01:57 -04:00
David Steele
3d07e01fe9 Make ls ordering deterministic in mock/all test. 2018-07-01 20:02:37 -04:00
Andrew Schwartz
1bd98b61df Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.
AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.

Fixed by Andrew Schwartz.
2018-07-01 08:17:27 -04:00
David Steele
7b0e65d488 Improve the HTTP client to set content-length to 0 when not specified by the server.
S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.

Suggested by Adam K. Sumner.
2018-06-26 17:27:22 -04:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
a385cb520b Update primary test environment (Vagrant and Docker) to Ubuntu 18.04. 2018-06-06 15:52:28 -04:00
David Steele
d309a85b51 PostgreSQL 11 Beta 1 support. 2018-06-05 08:59:17 -04:00
David Steele
e57840d739 Document generator improvements.
* Allow parameters to be passed when a container is created.
* Allow /etc/hosts update to be skipped (for containers without bash).
2018-05-25 13:42:09 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
8eb9921c7c Auto-generate Makefile with dependencies. 2018-05-23 14:57:08 -04:00
David Steele
f7123c1bbf v2.03: Single Executable to Deploy
Bug Fixes:

* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)

Improvements:

* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c Test harness improvements.
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
91be372e6a Set log-timestamp=n for integration tests.
This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 11:24:38 -04:00
David Steele
a1be8e16cd Update SSL error message test on CentOS 7. 2018-05-11 10:39:36 -04:00
David Steele
790f7c7312 Enforce full branch coverage in C code. 2018-05-05 11:50:11 -04:00
David Steele
c3a8fbe706 Add _FMT variants for all THROW macros so format types are checked by the compiler. 2018-05-03 11:24:29 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
5e090ba305 Fix failure in manifest build when two or more files in PGDATA are linked to the same directory.
Reported by Vitaliy Kukharik.
2018-05-02 12:19:54 -04:00
David Steele
8dbbf2de92 Fix archive-copy throwing "path not found" error for incr/diff backups.
Reported by yummyliu, Vitaliy Kukharik.
2018-05-01 12:37:06 -04:00
David Steele
71ba08f579 Use path list in the backup manifest to do restore path syncs.
Remove recursive path sync functionality since it is no longer used.
2018-05-01 11:05:37 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
David Steele
88a633da17 Divide tests into three types (unit, integration, performance).
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.

Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 09:12:25 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
cf0ec44ede Rename Perl tests so they don't conflict with their C counterparts. 2018-04-17 12:03:03 -04:00
David Steele
dce11ce3cf Generate code counts for all source files.
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00
David Steele
3e22977484 Use pip 9.03 in test VMs.
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
2018-04-16 15:25:50 -04:00
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier.
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module.
These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all.
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 12:16:32 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml.
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 10:19:24 -04:00
David Steele
b77b425096 Allow -DDEBUG_UNIT to be suppressed to test how debug macros behave. 2018-04-07 19:44:00 -04:00
David Steele
82751b3b51 Split debug and assert code into separate headers.
Assert can be used earlier because it only depends on the error-handler and not logging.
2018-04-07 13:12:35 -04:00
David Steele
e00f2dd4ad More C type cleanup.
* Remove typec.h entirely.
* Order all typdefs above local includes.
2018-04-06 21:46:45 -04:00
David Steele
f3b56cf8f3 Allow MemContext objects to be copied to a new parent.
This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.
2018-04-06 17:04:52 -04:00
David Steele
b9f636cab4 Cleanup C types.
* Remove pgBackRest defined types included with the C types.
* Remove redefinition of standard C types.
2018-04-05 22:38:11 -04:00
David Steele
bd25248df0 Abstract Posix storage driver code into a separate module. 2018-04-05 14:48:16 -04:00
David Steele
93fdb98d15 Storage object improvements.
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
David Steele
635caff573 Add ASSERT() that is preserved in production builds. 2018-03-30 19:10:34 -04:00
David Steele
c1ab7b3c98 Start work on C handle io object and use it to output help. 2018-03-25 11:39:24 -04:00
David Steele
dabf727d91 Include VM type in gcov path to avoid conflicts between VMs with different architectures. 2018-03-21 16:08:08 -04:00
David Steele
07f38f584a Use lcov for C unit test coverage reporting.
Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov.

Branch coverage is not complete, so for the time being errors will only be generated when statement coverage is not complete. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
2018-03-19 23:33:28 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present.
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
2018-03-19 19:54:03 -04:00
David Steele
0c313713b1 Use clang for static code analysis during lint testing.
Nothing found except for some functions that should have been marked __noreturn__.
2018-03-18 13:32:19 -04:00
David Steele
f0451c1494 Enable -Wformat=2 and -Wformat-nonliteral and silence new warnings. 2018-03-15 13:53:03 -04:00
David Steele
dd3ce70810 Config test code writes secure options to a file instead of passing on the command-line. 2018-03-15 11:09:46 -04:00
David Steele
4b6cd92a4c Improve debugging.
* Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT.
* Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.
2018-03-12 17:15:14 -04:00
David Steele
cced6ec03a Add ASSERT_DEBUG() macro for debugging.
Replace all current assert() calls except in tests that can't use the debug code.
2018-03-12 14:31:22 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
94494a632f Update pip before installing awscli. 2018-03-12 11:02:46 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
919635b6f4 Set config before Main::main() call to avoid secrets being exposed in a stack trace. 2018-03-08 16:24:16 -05:00
David Steele
e331564c90 Add new test for Common::Io::Process.
Show that output on stderr will raise an exception on close() even if the exit code is 0.
2018-03-08 12:36:55 -05:00
Cynthia Shang
45ffe37e4f Add cfgOptionTest().
Update cfgOption() calls that are better implemented as cfgOptionTest().

Contributed by Cynthia Shang.
2018-03-05 18:51:48 -05:00
David Steele
406acf9cba Enable -Wconversion and silence new warnings. 2018-03-02 12:07:12 -05:00
David Steele
5b491b3035 Add CentOS/RHEL package builds. 2018-02-26 19:32:27 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
David Steele
9070325760 Improve performance of HTTPS client.
Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
2018-02-18 14:53:27 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp().
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
5f2884cb29 Suppress coverage failures for Archive/Push/Async on Travis.
The coverage report shows some code as never being run -- but that makes no sense because the tests pass.  This may be due to trying to combine the C and Perl coverage reports and overwriting some runs.

Suppress for now with a plan to implement LCOV for the C unit tests.
2018-02-14 16:46:52 -05:00
David Steele
a907fd7d2d Build C binary/library for C unit tests that require Perl. 2018-02-14 10:33:21 -05:00
David Steele
fda62dfc48 Move Perl option JSON generation to a new module.
Linking exec.c into LibC will not be an option once the perl exec loads the Perl interpreter.
2018-02-14 09:49:01 -05:00
David Steele
5242e8eec5 Generate suppressions when running Valgrind.
This makes it easier to suppress errors that are only occurring on Travis.
2018-02-14 09:18:27 -05:00
David Steele
305a3e1761 Add int64 variant type. 2018-02-09 13:52:02 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml.
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
eba31fbfe5 Update URL for Debian package repository. 2018-02-08 14:49:35 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
c3f47bf240 Move config parsing out of Perl tests.
cfgParseTest() is provided in the C library for parsing configs in unit tests.
2018-02-05 12:32:30 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists.
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
Cynthia Shang
bacc08792d Ensure latest db-id is selected on when matching archive.info to backup.info.
This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade).

Fixed by Cynthia Shang.
Reported by Adam K. Sumner.
2018-01-30 16:49:47 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include.
Fixed by Cynthia Shang.
2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl.
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
f0ef1dee05 Improve speed of C unit tests.
Preserve object files between tests and use a Makefile to avoid rebuilding object files.
2018-01-26 16:41:17 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
404ec30d03 Move Perl help test to real module. 2018-01-23 12:58:35 -05:00
David Steele
2afb73da1c Improve String, StringList, and List objects.
* Add strCmp*() and strFirst*() to String.
* Add strLstSort() and strLstNewSplitSize() to StringList.
* Add strLstNewSplitZ() to StringList a update calls to strLstNewSplit() as needed.
* Add lstSort to List.
2018-01-23 12:17:30 -05:00
David Steele
2cc9b2287b The archive-push command is now partially coded in C.
This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
David Steele
7c73f4f6c9 Add command and exit modules. 2018-01-17 09:15:51 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
eb452c8add Improvements to Storage object.
* storagePath() and storageList() commands
* Helper functions for local and spool storag
2018-01-16 13:38:37 -05:00
David Steele
8f81620b9f Improve error module.
Add functions to convert error codes to C errors and handle system errors.
2018-01-16 13:29:27 -05:00
David Steele
d633417e5d Add Wait object and time module. 2018-01-16 13:05:00 -05:00
David Steele
721d1127a0 Add RegExp object. 2018-01-16 12:57:23 -05:00
David Steele
83ddd75d82 Improvements to String object.
* Add strBeginsWith(), strEndsWith(), strEq(), and strBase().
* Enable compiler type checking for strNewFmt() and strCatFmt().
* Rename strNewSzN() to strNewN().
2018-01-16 12:47:08 -05:00
David Steele
d992851456 Rename Perl tests so they don't conflict with their C counterparts. 2018-01-16 12:36:22 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
d8c575801d Add Ini object. 2017-12-22 22:55:17 -05:00
David Steele
69488ca8ff Add Storage object. 2017-12-22 22:36:01 -05:00
David Steele
b2a64b1f43 Add KeyValue object. 2017-12-22 22:05:37 -05:00
David Steele
2add6cef95 Add Variant and VariantList objects. 2017-12-22 21:50:27 -05:00
David Steele
c4b30af645 Add Buffer object. 2017-12-22 19:56:01 -05:00
David Steele
f89b027dec Improve String and StringList modules. 2017-12-22 19:30:15 -05:00
David Steele
5ea1263d8e Run valgrind on all C unit tests.
Also refactor code to make valgrind happy.
2017-12-22 18:36:36 -05:00
David Steele
9fe265d24c No longer run master branch through CI.
The integration branch will be run through CI and then pushed to master with github status checks.
2017-12-19 17:03:17 -05:00
Cynthia Shang
c34a76690f Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster.
Fixed by Cynthia Shang.
Reported by Stephen Frost.
2017-12-19 15:39:39 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10.
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built.
Contributed by Cynthia Shang.
2017-12-13 11:16:27 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated.
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore.
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
59fd92fdd5 Move restore test infrastructure to HostBackup.pm.
Required to test restores on the backup server, a fairly common scenario.

Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-03 17:08:49 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
Cynthia Shang
bd74711ceb Add unit tests for the Manifest module.
Also minor changes to Manifest module, mostly for test reproducibility.

Contributed by Cynthia Shang.
2017-11-28 11:44:24 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl.
This allows certain time-critical commands (like async archive-push) to run more quickly.
2017-11-26 18:43:51 -05:00
David Steele
4d8ad4ac18 Add List and StringList objects to simplify list handling. 2017-11-26 18:23:11 -05:00
David Steele
1b3e8e0a5b Add String object to simplify string handling. 2017-11-26 17:49:10 -05:00
David Steele
7cf955425e The C library is now required.
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
ac54aac448 Fixed an issue passing parameters to remotes.
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
2017-11-21 13:17:44 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function.
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0.
The filter was used with compress level set to 0 which added overhead without any benefit.
2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore.
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 12:46:35 -05:00
David Steele
8ccfaa9ad3 Designate a single distro (Ubuntu 16.04) for C coverage testing
Enable optimizations on the distros that don't do coverage testing.
2017-11-20 09:05:55 -05:00
David Steele
3b43047e0d Disable -Wclobber compiler warning.
...because it is mostly useless but keep the rest of of -Wextra.
2017-11-19 16:44:33 -05:00
David Steele
500d6b4b66 Automated tests for 32-bit i386/i686 architecture. 2017-11-18 20:14:14 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files.
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-18 20:02:54 -05:00
David Steele
b701cd5a5f Designate a single distro (Ubuntu 16.04) for Perl coverage testing.
Running coverage testing on multiple distros takes time but doesn't add significant value. Also ensure that the distro designated to run coverage tests is one of the default test distros.
2017-11-18 19:53:41 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions.
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test.
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
2017-11-15 13:17:57 -05:00
David Steele
22fe684b1f Only check expect logs on CentOS 7.
Variations in distros cause false negatives in tests but don't add much value.
2017-11-14 17:16:39 -05:00
David Steele
b8746f368d Inflate performance improvement for gzip filter and full unit test coverage. 2017-11-14 15:12:31 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore.
Reported by Craig A. James.
2017-11-14 14:48:44 -05:00
David Steele
bcdfc7d0b5 Enable additional warnings for C builds. 2017-11-06 22:55:34 -05:00
Cynthia Shang
b03c26968a Repository encryption support.
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
David Steele
8d6a08a32b Library code for repository encryption support. 2017-11-03 13:57:58 -04:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
f57e376c44 Update Debian/Ubuntu containers to download lastest version of pip. 2017-10-30 09:29:46 -04:00
David Steele
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored.
Reported by Jens Wilke.
2017-10-22 18:05:46 -04:00
David Steele
3553ccae44 Add I/O performance tests. 2017-10-22 13:27:47 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
f4524aeaa9 Add base64 encode/decode. 2017-10-18 11:14:26 -04:00
David Steele
b1efd598cf Add PostgreSQL versions to Debian VMs for testing. 2017-10-16 18:21:47 -04:00
David Steele
c94e52f697 Remove error when overlapping timelines are detected.
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.

Reported by blogh.
2017-10-16 14:34:44 -04:00
David Steele
583a76f605 Add C memory contexts. 2017-10-16 11:25:49 -04:00
David Steele
6f5186f9e6 Convert config tests into C unit tests. 2017-10-16 11:13:51 -04:00
David Steele
9d78948a14 Improvements to page checksum module and tests:
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
2017-10-16 11:03:06 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler.
Basic try ... catch ... finally pattern to simplify error handling in C.
2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5 Add C unit test infrastructure. 2017-10-12 12:55:48 -04:00
David Steele
abea4d1fd5 Remove Debian test repo after PostgreSQL 10 release. 2017-10-05 12:33:13 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
7c95a2f4de Updates to installation documentation.
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
2017-09-08 21:50:00 -07:00
David Steele
6343fdd584 Additional backup exclusions.
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 08:26:57 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways.
Suggested by Victor Gdalevich.
2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.`
Contributed by Cynthia Shang.
2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files.
Reported by Jesper St John, Aleksandr Rogozin.
2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test.
Contributed by Cynthia Shang.
2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade.
Fixed by Cynthia Shang.
2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
fa992433da Update CI code to prevent deprecated VM from being rewritten. 2017-07-27 11:31:42 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups.
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements:
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
2017-06-24 10:59:00 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
6327799c9a Use old trusty image until new image works with RHEL6. 2017-06-22 18:13:59 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00