1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1232 Commits

Author SHA1 Message Date
David Steele
1b3770e248 Recopy during backup when resumed file is missing or corrupt.
A recopy would occur if the size or checksum was invalid but on error the backup would terminate.

Instead, recopy the resumed file on any error.  If the error is systemic (e.g. network failure) then it should show up again during the recopy.
2019-12-07 09:48:33 -05:00
David Steele
e632c60525 Fix backup labels in mock/all resume integration tests.
These were not getting updated to match the directory name when the manifests were copied.

The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 11:48:41 -05:00
David Steele
b2d82bd248 Add functions to get the substring found by regExpMatch().
For now this is only used in testing but there are places where it could be useful in the core code.

Even if that turns out not to be true, it doesn't seem worth implementing a new version in testing just to capture a few values that we already have.
2019-12-04 19:43:26 -05:00
David Steele
8dfe0e48e2 Use more general error code when tablespace linked into PGDATA.
The specific error code was not that useful since we also test the error message which contains details of the link error.
2019-12-02 10:49:25 -05:00
David Steele
33a63aae50 Add flag to dbGet() to require a standby.
This is needed from backup from standby functionality.
2019-12-02 07:39:42 -05:00
David Steele
28116918ff Error in remote command when stop file exists.
This duplicates the Perl functionality.
2019-12-02 07:35:36 -05:00
David Steele
fc291b6f28 Reduce the scope of mock/all exclusion tests.
Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-01 17:47:47 -05:00
David Steele
d15ed33821 Make MCV return false when a boolean tie.
This is to maintain compatibility with the older Perl code that returned the lowest sorted order item in a tie.

For other datatypes the C code returns the same value, often enough at least to not cause churn in the expect tests.
2019-12-01 16:32:21 -05:00
David Steele
56ee321a95 Add pgLsnName() and pgXactPath(). 2019-12-01 15:49:34 -05:00
David Steele
50eb062e0e Fix reference list when backup.info is reconstructed in expire command.
Adding a manifest to backup.info was migrated to C in 4e4d1f41 but deduplication of the references was missed leading to a reference for every file being added to backup.info.

Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual.

This unlikely combination of events means this is probably not a problem in the field.
2019-11-28 09:34:19 -05:00
David Steele
686b6f91da Set archive-check option in manifest correctly when offline.
Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 08:27:21 -05:00
David Steele
158e439689 Remove obsolete Perl archive code.
This should have been removed in a1c13a50 but was missed.
2019-11-26 17:16:45 -05:00
David Steele
82df7e6f3b Update integration tests in real/all that use test points.
Test points are not supported by the new C code so these will be replaced with unit tests.

The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place.

Even so, this does represent a regression in (soon to be be removed) Perl coverage.
2019-11-26 11:32:12 -05:00
David Steele
b145c72b5c Update missing manifest warning in BackupInfo.
This brings the Perl message in line with C to reduce expect log churn.
2019-11-25 08:51:28 -05:00
David Steele
8800f32ad9 Remove exclusions once they have been tested in mock/all.
The exclusions no longer have any effect after a restore and just add noise to the expect log.
2019-11-25 08:35:26 -05:00
David Steele
9856fef586 Update integration tests in mock/all that use test points.
Test points will not be available in the C code so update these tests as best as possible without using them.

This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.
2019-11-25 07:48:52 -05:00
David Steele
3cd45a7411 Remove start/stop --force integration tests in mock/all.
These tests require test points which are not being implemented in the C code.

This functionality is fully tested in the command/control unit tests so integration tests are no longer required.
2019-11-25 07:45:58 -05:00
David Steele
01aefc563d Update Perl page checksum expression.
This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.

However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.

Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 07:37:09 -05:00
David Steele
cace54151f Add hostId to protocolLocalGet().
Previously this function was only creating locals that talked to the repository.  Backup will need to be able to talk to multiple PostgreSQL hosts.
2019-11-23 10:32:57 -05:00
David Steele
ab65ffdfac Add protocolStorageType*() to manage protocol storage types.
Abstract the string representation of storage types that are passed over the protocol layer.
2019-11-23 10:22:11 -05:00
David Steele
a4b9440d35 Only install specific lcov version when required.
Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.

Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.

PostgreSQL minor version releases are also included since all containers have been rebuilt.
2019-11-22 19:25:49 -05:00
David Steele
52a3ba6b6f Revert "Forbid % character in parameters."
The issue "fixed" in f01aa586 was caused by treating all strings as format strings while logging, which was fixed in 0c05df45.

Revert because there no longer seems a reason for the extra logic, and it was only partially applied, i.e. not to env vars, command-line options, or config options.
2019-11-22 15:18:56 -05:00
David Steele
381aecae4e Fix walPath() when CWD is / and path is relative.
The function would return a // prefix in this case, which works fine but looks odd while debugging.
2019-11-22 14:30:56 -05:00
David Steele
0c05df4582 Add _FMT() logging macro variants.
Using the same macros for formatted and unformatted logging had several disadvantages.

First, the compiler was unable to verify the format string against the parameters.

Second, legitimate % characters in messages were being interpreted as format characters with garbage output ensuing.

Add _FMT() variants and update all call sites to use the correct variant.
2019-11-22 13:33:26 -05:00
David Steele
f01aa5861d Forbid % character in parameters.
This character causes problems in C and in the shell if we try to output it in an error message.

Forbid it completely and spell it out in error messages to avoid strange effects.

There is likely a better way deal with the issue but this will do for now.
2019-11-21 17:28:03 -05:00
David Steele
c524ec4f95 Remove obsolete integration tests from mock/all.
The protocol timeout tests have been superceded by unit tests.

The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto.  It didn't hurt anything but it did add 200ms to each test where it appeared.

Catalog and control version tests were redundant.  The database version and system id tests covered the important code paths and the C code gets these values from a lookup table.

Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 16:06:27 -05:00
David Steele
270f9496e4 Add manifestMove(). 2019-11-21 12:08:32 -05:00
David Steele
c5a6631d27 Rearrange manifest module.
Put functions with related functions, move getters above the helper functions, and rename manifestPgPath() to manifestPathPg().
2019-11-21 11:44:40 -05:00
David Steele
9f71a019c8 Allow storageInfo() to operate outside the base storage path.
It is occasionally useful to get information about a file outside of the base storage path.  storageLocal() can be used in some cases but when the storage is remote is doesn't seem worth creating a separate storage object for adhoc info requests.

storageInfo() is a read-only operation so this seems pretty safe.  The noPathEnforce parameter will make auditing exceptions easy.
2019-11-21 10:55:03 -05:00
David Steele
d3b1897625 Allow adhoc enforcement in storagePath().
The ability to disable enforcement (i.e., the requested absolute path is within the storage path) globally will be removed after the Perl migration.

The feature will still be needed occasionally so allow it in an adhoc fashion.
2019-11-21 10:34:32 -05:00
David Steele
e1dad720a1 Rename storagePath() to storageP() in places where it was missed.
Correct this since it will be enforced in a subsequent patch.
2019-11-21 10:21:35 -05:00
David Steele
cef9f0f37f Process . in strPathAbsolute().
A . in a link will always lead to an error since the destination will be inside PGDATA.  However, it is accepted symlink syntax so it's better to resolve it and get the correct error message.

Also, we may have other uses for this function in the future.
2019-11-21 09:40:15 -05:00
David Steele
63c4c14836 Fix lcov build in Vagrantfile.
-q was being instead of -s for silent mode which caused the build to fail.
2019-11-19 20:52:01 -05:00
David Steele
1db9e3b144 Remove *MP() macros variants.
Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.

-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
David Steele
09e129886e Add storageInfoList() support to remote storage driver. 2019-11-16 17:47:42 -05:00
David Steele
26e1da82e7 Allow zero-length substrings to be extracted from the end of a string.
The previous assert was a bit overzealous and did not allow this case.  It's not very common but still occasionally useful.
2019-11-16 17:32:49 -05:00
David Steele
8a3de1e05a Add storageInfo() support to remote storage driver. 2019-11-16 17:30:08 -05:00
David Steele
8d6a8c3bf0 Store base path for remote storage locally.
It wasn't practical for the main process to be ignorant of the remote path, and in any case knowing the path makes debugging easier.

Pull the remote path when connecting and pass the result of local storagePath() to the remote when making calls.
2019-11-16 17:12:16 -05:00
David Steele
6827a13f3a Add facility for reading and writing adhoc protocol output.
Pushing output through a JSON blob is not practical if the output is extremely large, e.g. a backup manifest with 100K+ files.

Add read/write routines so that output can be returned in chunks but errors will still be detected.
2019-11-16 17:05:34 -05:00
David Steele
c8db11e65b Add user-id/group-id to hrnReplaceKey(). 2019-11-15 17:50:12 -05:00
David Steele
53a2d04ab0 Allow "null" in jsonToStr(). 2019-11-15 17:48:25 -05:00
David Steele
48e8942e86 Allow trailing / for relative paths in strPathAbsolute().
The trailing / does nothing but is nevertheless valid syntax.
2019-11-15 08:53:15 -05:00
David Steele
3b879c2cb3 Filter logged command options based on the command definition.
Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.

Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.
2019-11-14 16:48:41 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
David Steele
a44c5d0315 Add Strings for STORAGE_REPO_ARCHIVE and STORAGE_REPO_BACKUP.
These constants are used often enough that they deserve to have String constants rather than repeatedly calling STRDEF().
2019-11-12 13:12:07 -05:00
David Steele
10c8eeaf6c Fix handling of repeated HTTP headers.
When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior.

Reported by donicrosby.
2019-11-08 18:58:45 -05:00
David Steele
4317178633 Update MinIO to newest release.
We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.

In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.
2019-11-08 17:56:34 -05:00
David Steele
edcc7306a3 Add TIME parameter debug type.
Previously we were using int64_t to debug time_t but this may not be right depending on how the compiler represents time_t, e.g. it could be a float.

Since a mismatch would have caused a compiler error we are not worried that this has actually happened, and anyway the worst case is that the debug log would be wonky.

The primary benefit, aside from correctness, is that it makes choosing a parameter debug type for time_t obvious.
2019-11-08 09:46:00 -05:00
David Steele
8b682b75d2 Allow mock integration tests for all VM types.
Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 10:35:48 +01:00
David Steele
7168e07440 Use getcwd() to construct path when WAL path is relative.
Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places.  PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00