1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1766 Commits

Author SHA1 Message Date
David Steele
ec347847e5 Pass cipher type directly to backupFileProtocol().
Cipher type was inferred from the presence of cipherSubPass rather than being passed explicitly in order to maintain compatibility with Perl backupFile().

Now that Perl is gone it makes sense to pass it explicitly, as we do elsewhere.
2021-03-12 15:19:32 -05:00
David Steele
e07040c2e4 Add HRN_STORAGE_TIME() harness macro.
Makes updating the time of a path/file more streamlined in tests.

Also update all tests where utime() was being used directly.
2021-03-12 12:54:34 -05:00
David Steele
3c85a497a6 Add backup delta unit test.
This test was added to take the place of another test, which turned out not to be workable.

Even so, it adds coverages at little cost so it seems worth keeping.
2021-03-11 14:40:14 -05:00
David Steele
dc1052f1da Remove extra spaces before macro continuation. 2021-03-11 14:11:21 -05:00
David Steele
c862e9654a
Log archive copy during backup.
Copying can be a fairly expensive operation so it makes sense to log it so the user gets some status during long copy operations.
2021-03-11 08:22:44 -05:00
David Steele
28301199eb Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*().
When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed.

Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
Cynthia Shang
31c7824a4d
Allow stanza-* commands to be run remotely.
The stanza-create, stanza-upgrade and stanza-delete were required to be run on the repository host. When there was only one repository allowed this was not a problem.

However, with the introduction of multiple repository support, this becomes more of a burden to the user, therefore the stanza-create, stanza-upgrade and stanza-delete commands have been improved to allow for them to be run remotely.
2021-03-10 08:10:46 -05:00
David Steele
dde2e2326b Print module type (e.g. c or h) in stack trace.
Now that there are inline functions in .h files it is important to include the extension so the functions can be found when debugging.
2021-03-08 17:32:36 -05:00
David Steele
fe4ba455ed Move configuration definition to src/build/config/config.yaml.
Moving to YAML allows the configuration data to be read by C programs.

Also go back to using YAML::XS since it is the only implementation that has proper boolean support.
2021-03-08 16:01:05 -05:00
David Steele
1dbb3bf50b
Multiple repository support.
Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.

Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.

Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.

The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
For single repository configurations there should be no change in behavior.
2021-03-08 13:31:13 -05:00
David Steele
a5f07dff0a Remove autoconf cache when configure is built. 2021-03-05 16:27:57 -05:00
David Steele
088662d986
GCS support for repository storage.
GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
David Steele
95063f6812 Make --repo optional for remaining commands except stanza-delete.
Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified.

This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.
2021-03-03 09:21:06 -05:00
David Steele
d1aa765a9d
Consolidate less commonly used repository storage options.
The following options are renamed as specified:

repo1-azure-ca-file -> repo1-storage-ca-file
repo1-azure-ca-path -> repo1-storage-ca-path
repo1-azure-host -> repo1-storage-host
repo1-azure-port -> repo1-storage-port
repo1-azure-verify-tls -> repo1-storage-verify-tls
repo1-s3-ca-file -> repo1-storage-ca-file
repo1-s3-ca-path -> repo1-storage-ca-path
repo1-s3-host -> repo1-storage-host
repo1-s3-port -> repo1-storage-port
repo1-s3-verify-tls -> repo1-storage-verify-tls

The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.
2021-03-02 13:51:40 -05:00
David Steele
e64999db77
Add HttpUrl object.
Parse a URL into component parts.
2021-03-01 13:44:47 -05:00
David Steele
a1341b4af0 Make S3/Azure file missing error messages match Posix.
The S3 driver was missed when the constants were added and then Azure was copied from S3.
2021-02-28 17:00:41 -05:00
David Steele
3b8f0ef7ae Add write fault-tolerance to archive-push command.
The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.
2021-02-26 16:52:59 -05:00
David Steele
a1280c41e5 Refactor archive-push command warnings to work like archive-get.
Warnings are logged individually in the async log rather than all together.
2021-02-26 15:58:11 -05:00
Cynthia Shang
13dc8e68d7 Make --repo optional for backup command.
If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 14:49:50 -05:00
Cynthia Shang
0ddc0380ff Remove restore default repo from integration tests.
The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 11:32:13 -05:00
David Steele
8f03c3574b Reduce default file log level for integration tests.
The real/all test could fill the ramdisk depending on which vm and pg version were selected.

Debug level should be fine for most purposes and the level can be increased when needed.
2021-02-24 08:27:58 -05:00
Cynthia Shang
118d9e64fe Enhance restore command multi-repo support.
The restore command automatically defaults to selecting the latest backup from a single repository. With multiple repositories configured, the restore command will now default to selecting the latest backup from the first repository where backups exist. The order in which the repositories are checked is dictated by the pgbackrest.conf order.

To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
2021-02-23 16:17:27 -05:00
David Steele
bec3e20b2c Add archive-get command multi-repo support.
Repositories will be searched in order for the requested archive file.

Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Cynthia Shang
e28f6f11e9 Expire continues if an error occurs processing a repository.
Errors are logged to the log file rather than thrown. If, after processing all repos, one or more errors occurred, then a single error error will be thrown to indicate there were errors and the log file should be inspected.

Also update log messages to be more consistent with new patterns.
2021-02-23 12:20:02 -05:00
David Steele
00b60e564e Add base64url encoding.
For now only encoding is supported. Decoding is not needed and may never be.
2021-02-19 19:21:06 -05:00
David Steele
f6c3262861 Do not expose valid/validate functions from encode module.
These functions have never been used externally. Validation is always part of decoding so performing validation separately would be wasteful.
2021-02-19 18:25:50 -05:00
David Steele
edab2a0b89 Use switch rather than if-else for encoding types.
This is more efficient and the error case can be an assert rather than a runtime error.

For extra safety initialize destinationSize to SIZE_MAX to increase the chances of an error if the switch fails.
2021-02-19 17:57:13 -05:00
David Steele
abcbe0f9c1 Combine encode module files into a single file.
There is not enough code here to justify multiple files and declaring the functions for each encoding as static allows the compiler to inline where appropriate.
2021-02-19 17:25:00 -05:00
David Steele
d485609658 Add strNewEncode(), strCatEncode(), and bufNewDecode().
These constructors wrap encodeToStr() and decodeToBin(), making them convenient and safe by eliminating the need to create intermediate buffers. Encoding/decoding is performed directly into the target String/Buffer. Sizing of the destination buffer is handled by the new functions so it doesn't have to be done at each call site.
2021-02-19 17:05:15 -05:00
David Steele
5b98968605 Do not lower-case help summaries when first word is an acronym.
If the second letter is capital or a digit then the word is likely an acronym so don't lower-case the first letter.

For now only the digit case is checked since there are no summaries with a capital as the second letter.
2021-02-19 10:29:29 -05:00
David Steele
66a4ff496a Encode path before passing to HttpRequest.
GCS requires mixed encoding in the path so encoding inside HttpRequest does not work.

Instead, require the path to be correctly encoded before being passed to HttpRequest.
2021-02-19 09:05:32 -05:00
David Steele
1b4b3538cc Rename uri to path where appropriate in HTTP and storage modules.
The path was originally named uri due to the canonicalized path being called "canonicalized uri" in the S3 authentication documentation. The name got propagated everywhere from there.

This is not correct for general usage, however, so rename to path when describing the path component of an HTTP request.
2021-02-19 08:22:50 -05:00
David Steele
dcb79ab8fb Decode JSON \u escaped characters.
ASCII may occasionally be encoded (e.g. &) to prevent ambiguity depending on where the JSON is located.

Only ASCII can be decoded. In general Unicode should not be encoded in JSON.
2021-02-19 07:32:40 -05:00
David Steele
c4243331de Silence chmod warning of empty path when removing unit test data. 2021-02-19 07:24:59 -05:00
David Steele
3837e61a75
Fix option warnings breaking async archive-get/archive-push.
Option warnings will cause the async process to fail because a warning is logged but stdout is closed so the process aborts.

This bug has existed for quite some time, but it was made worse by abb8ebe because now the async role can have different valid options than the default role. Previously at least a warning would be emitted before the async process died.

Fix this by only allowing warnings for the default role. Warnings were already suppressed for local and remote roles so the logic already exists.
2021-02-18 13:29:09 -05:00
David Steele
d7befd4189 Fix tests that ensure log levels are not set for local/remote roles.
These tests were broken because they were being gated by resetLogLevel. So they were not setting the log levels, but not because of the role setting. Because resetLogLevel was being checked last coverage testing indicated that the tests were working.

Fix the resetLogLevel parameter in the tests and move resetLogLevel to be tested first so coverage reporting works as expected. This isn't perfect but it is an improvement.
2021-02-16 16:28:19 -05:00
David Steele
920c746adb Better exclusions for configure help.
Exclude known unused options rather than trying to include used options. This works better when new options are added.
2021-02-10 15:54:37 -05:00
Cynthia Shang
3408f1ee2e Enhance expire command multi-repo support.
The expire command has been enhanced to expire backups and archives from all configured repositories by default.

In addition, it will accept the --repo option to expire backups and archives only from the specified repository. Using the --repo options the --set option can also be refined further to the specified repo. If --set is provided but the --repo option has not, then all repositories will be searched and retention settings will be applied on each whether the backup set has been found or not.
2021-02-10 12:03:52 -05:00
David Steele
c7d7280fa8 Ensure test user has permissions before removing test files.
This allows files to be deleted even when tests have limited the permissions.
2021-02-08 16:36:38 -05:00
David Steele
b2bba678a0 Add missing linefeeds. 2021-02-08 16:28:16 -05:00
David Steele
aadc9e2fe6 v2.32: Repository Commands
Bug Fixes:

* Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)

Features:

* Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)
* Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.)

Improvements:

* Improve archive-get performance. (Reviewed by Cynthia Shang.)
2021-02-08 09:08:16 -05:00
David Steele
b65c370346 Add repo-get command. 2021-02-05 10:39:03 -05:00
David Steele
218cd078a6 Add repo-ls command. 2021-02-05 10:07:43 -05:00
David Steele
9154d73030 Add -g accidentally removed in 4e8d469f.
The tests all run fine without debug info but gdb and valgrind are a lot less useful without it.
2021-02-02 17:05:55 -05:00
Stefan Fercot
4b46115345
Add archive-mode-check option.
This option disallows the PostgreSQL archive_mode=always setting and disabling it allows the setting.
2021-02-02 13:43:14 -05:00
David Steele
101bf5d114 Log configuration parameters in config test harness.
This makes it easier to find the current configuration in tests.
2021-02-01 14:50:22 -05:00
David Steele
9baf5d7da7 Set dry-run before updating options in config test harness.
This is the same order that is used in cfgLoad() so apply it here as well.

Also remove vestigial HRN_FEATURE_STORAGE define.
2021-02-01 14:42:22 -05:00
David Steele
67d444b9e8 Add bufEmpty().
This seems more readable than bufUsed() == 0, just like 7d6c0319.
2021-02-01 09:22:01 -05:00
David Steele
7d6c0319f0 Add lstEmpty(), strLstEmpty(), and varLstEmpty().
This seems more readable than lst*Size() == 0.

Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.
2021-01-29 14:27:56 -05:00
David Steele
5281e31422 Add configurable error handlers.
The stackTrace and memContext error handlers were hard-coded which made testing the error module in isolation impossible.

Making the error handlers configurable also makes adding new ones in the future easier.
2021-01-27 17:25:13 -05:00