1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

601 Commits

Author SHA1 Message Date
David Steele
e206093beb Allow end anchor to be excluded in backupRegExp().
This is useful for matching files in the backup history directory which have characters after the backup label.
2019-12-12 18:52:16 -05:00
David Steele
8acfb6adf4 Add pgLsnRangeToWalSegmentList() to convert lsn range to wal segments. 2019-12-12 16:43:34 -05:00
David Steele
81295fd388 Move not found error into walSegmentFind().
This error is also needed in backup so move it here to centralize it.
2019-12-12 16:28:26 -05:00
David Steele
1378d9c58b Fix bad arithmetic in pgLsnToWalSegment().
/ takes precedence over & but the appropriate parens were not provided.

By some bad luck the tests worked either way, so add a new test that only works the correct way to prevent a regression.
2019-12-12 16:21:51 -05:00
David Steele
676be2c773 Add pgWalPath() to return version-specific WAL path.
Also update the manifest module to use the new function.
2019-12-12 16:11:09 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
0194a98671 Fix archive-push/archive-get when PGDATA is symlinked.
Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked.

If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
David Steele
8c840c28a6 Fix segfault on unexpected EOF in gzip decompression.
If the compressed stream terminated early then the decompression process would get a flush request (NULL input buffer) since the filter was not marked as done.  This could happen on a zero-length or truncated (i.e. invalid) compressed file.

Change the existing assertion to an error to catch this condition in production gracefully.
2019-12-11 08:48:46 -05:00
David Steele
c933f12f9c Remove obsolete --perl-option option.
This option was used when Perl was executed instead of being embedded.

It has been obsolete for a long time so remove it.
2019-12-10 13:28:15 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
800d2972b0 Remove stray uint type.
This was probably copied from an example but some compilers don't like it.
2019-12-09 18:28:20 -05:00
Cynthia Shang
ca33545630 Remove redundant test and move another test. 2019-12-09 14:06:32 -05:00
David Steele
d3132dae26 Add functions for building new manifests.
New manifests are built before a backup is performed.

Reviewed by Cynthia Shang.
2019-12-08 18:43:47 -05:00
David Steele
2cfde18755 Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment(). 2019-12-08 14:19:47 -05:00
David Steele
f517b141fb Update pq harness to play nicely with variable LSNs. 2019-12-08 14:15:23 -05:00
David Steele
d2587250da Add backup functions to Db object.
These functions implement the database backup functionality for all supported versions.
2019-12-07 18:44:06 -05:00
David Steele
8766326da8 Add protocolRemoteFree() to shutdown a specific remote.
Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
David Steele
35a262951a Pq test harness usability and error reporting improvements.
Pq script errors are now printed in test output in case they are being masked by a later error.

Once a script error occurs, the same error will be thrown forever rather than throwing a new error on the next item in the script.

HRNPQ_MACRO_CLOSE() is not required in scripts unless harnessPqScriptStrictSet(true) is called.  Most higher-level tests should not need to run in strict mode.

The command/check test seems to require strict mode but there's no apparent reason why it should.  This would be a good thing to look into at some point.
2019-12-07 17:33:34 -05:00
David Steele
d6479ddd0e Add log replacements to help test non-deterministic log output.
Some log output (e.g. time) is hard to test because the values can change between tests.

Add expressions to replace substrings in the log with predictable values to simplify testing.

This is similar to the log replacement facility available for Perl expect log testing.
2019-12-07 17:15:20 -05:00
David Steele
e4716ee036 Improve diff output in tests.
Always compare expected vs actual (in that order) and give a hint in the error message to indicate what should be added and what removed.
2019-12-07 17:02:41 -05:00
David Steele
8c47ee296a Improve storage harness test callback.
Add ability to omit the root (i.e. dot) path and get real size of compressed files.
2019-12-07 16:55:50 -05:00
David Steele
1b3770e248 Recopy during backup when resumed file is missing or corrupt.
A recopy would occur if the size or checksum was invalid but on error the backup would terminate.

Instead, recopy the resumed file on any error.  If the error is systemic (e.g. network failure) then it should show up again during the recopy.
2019-12-07 09:48:33 -05:00
David Steele
b2d82bd248 Add functions to get the substring found by regExpMatch().
For now this is only used in testing but there are places where it could be useful in the core code.

Even if that turns out not to be true, it doesn't seem worth implementing a new version in testing just to capture a few values that we already have.
2019-12-04 19:43:26 -05:00
David Steele
33a63aae50 Add flag to dbGet() to require a standby.
This is needed from backup from standby functionality.
2019-12-02 07:39:42 -05:00
David Steele
28116918ff Error in remote command when stop file exists.
This duplicates the Perl functionality.
2019-12-02 07:35:36 -05:00
David Steele
d15ed33821 Make MCV return false when a boolean tie.
This is to maintain compatibility with the older Perl code that returned the lowest sorted order item in a tie.

For other datatypes the C code returns the same value, often enough at least to not cause churn in the expect tests.
2019-12-01 16:32:21 -05:00
David Steele
56ee321a95 Add pgLsnName() and pgXactPath(). 2019-12-01 15:49:34 -05:00
David Steele
50eb062e0e Fix reference list when backup.info is reconstructed in expire command.
Adding a manifest to backup.info was migrated to C in 4e4d1f41 but deduplication of the references was missed leading to a reference for every file being added to backup.info.

Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual.

This unlikely combination of events means this is probably not a problem in the field.
2019-11-28 09:34:19 -05:00
David Steele
cace54151f Add hostId to protocolLocalGet().
Previously this function was only creating locals that talked to the repository.  Backup will need to be able to talk to multiple PostgreSQL hosts.
2019-11-23 10:32:57 -05:00
David Steele
ab65ffdfac Add protocolStorageType*() to manage protocol storage types.
Abstract the string representation of storage types that are passed over the protocol layer.
2019-11-23 10:22:11 -05:00
David Steele
52a3ba6b6f Revert "Forbid % character in parameters."
The issue "fixed" in f01aa586 was caused by treating all strings as format strings while logging, which was fixed in 0c05df45.

Revert because there no longer seems a reason for the extra logic, and it was only partially applied, i.e. not to env vars, command-line options, or config options.
2019-11-22 15:18:56 -05:00
David Steele
381aecae4e Fix walPath() when CWD is / and path is relative.
The function would return a // prefix in this case, which works fine but looks odd while debugging.
2019-11-22 14:30:56 -05:00
David Steele
0c05df4582 Add _FMT() logging macro variants.
Using the same macros for formatted and unformatted logging had several disadvantages.

First, the compiler was unable to verify the format string against the parameters.

Second, legitimate % characters in messages were being interpreted as format characters with garbage output ensuing.

Add _FMT() variants and update all call sites to use the correct variant.
2019-11-22 13:33:26 -05:00
David Steele
f01aa5861d Forbid % character in parameters.
This character causes problems in C and in the shell if we try to output it in an error message.

Forbid it completely and spell it out in error messages to avoid strange effects.

There is likely a better way deal with the issue but this will do for now.
2019-11-21 17:28:03 -05:00
David Steele
270f9496e4 Add manifestMove(). 2019-11-21 12:08:32 -05:00
David Steele
c5a6631d27 Rearrange manifest module.
Put functions with related functions, move getters above the helper functions, and rename manifestPgPath() to manifestPathPg().
2019-11-21 11:44:40 -05:00
David Steele
9f71a019c8 Allow storageInfo() to operate outside the base storage path.
It is occasionally useful to get information about a file outside of the base storage path.  storageLocal() can be used in some cases but when the storage is remote is doesn't seem worth creating a separate storage object for adhoc info requests.

storageInfo() is a read-only operation so this seems pretty safe.  The noPathEnforce parameter will make auditing exceptions easy.
2019-11-21 10:55:03 -05:00
David Steele
d3b1897625 Allow adhoc enforcement in storagePath().
The ability to disable enforcement (i.e., the requested absolute path is within the storage path) globally will be removed after the Perl migration.

The feature will still be needed occasionally so allow it in an adhoc fashion.
2019-11-21 10:34:32 -05:00
David Steele
e1dad720a1 Rename storagePath() to storageP() in places where it was missed.
Correct this since it will be enforced in a subsequent patch.
2019-11-21 10:21:35 -05:00
David Steele
cef9f0f37f Process . in strPathAbsolute().
A . in a link will always lead to an error since the destination will be inside PGDATA.  However, it is accepted symlink syntax so it's better to resolve it and get the correct error message.

Also, we may have other uses for this function in the future.
2019-11-21 09:40:15 -05:00
David Steele
1db9e3b144 Remove *MP() macros variants.
Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.

-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
David Steele
09e129886e Add storageInfoList() support to remote storage driver. 2019-11-16 17:47:42 -05:00
David Steele
26e1da82e7 Allow zero-length substrings to be extracted from the end of a string.
The previous assert was a bit overzealous and did not allow this case.  It's not very common but still occasionally useful.
2019-11-16 17:32:49 -05:00
David Steele
8a3de1e05a Add storageInfo() support to remote storage driver. 2019-11-16 17:30:08 -05:00
David Steele
8d6a8c3bf0 Store base path for remote storage locally.
It wasn't practical for the main process to be ignorant of the remote path, and in any case knowing the path makes debugging easier.

Pull the remote path when connecting and pass the result of local storagePath() to the remote when making calls.
2019-11-16 17:12:16 -05:00
David Steele
6827a13f3a Add facility for reading and writing adhoc protocol output.
Pushing output through a JSON blob is not practical if the output is extremely large, e.g. a backup manifest with 100K+ files.

Add read/write routines so that output can be returned in chunks but errors will still be detected.
2019-11-16 17:05:34 -05:00
David Steele
c8db11e65b Add user-id/group-id to hrnReplaceKey(). 2019-11-15 17:50:12 -05:00
David Steele
53a2d04ab0 Allow "null" in jsonToStr(). 2019-11-15 17:48:25 -05:00
David Steele
48e8942e86 Allow trailing / for relative paths in strPathAbsolute().
The trailing / does nothing but is nevertheless valid syntax.
2019-11-15 08:53:15 -05:00
David Steele
3b879c2cb3 Filter logged command options based on the command definition.
Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.

Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.
2019-11-14 16:48:41 -05:00