1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3316 Commits

Author SHA1 Message Date
David Steele
4d48850aad Update negate to indicate that it should not be used for new options. 2021-05-19 12:58:48 -04:00
David Steele
870bcf30af Add missing header.
This worked because all places where this header was included, common/type/stringId.h was included first.
2021-05-18 16:48:30 -04:00
David Steele
4fa95af447 Fix typo in struct typedef.
This worked because the type is never declared as struct ExecPub.
2021-05-18 16:38:57 -04:00
Cynthia Shang
c92bc43c1e Simplify defaults for --force option.
All commands have the same default so it is more efficient to set the default for the entire option.
2021-05-18 16:33:45 -04:00
David Steele
320c6e1aad
Remove stanza archive spool path on restore.
Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but better to be safe. Missing stanza spool paths are ignored.

Also add new path expression STORAGE_SPOOL_ARCHIVE to easily access this path.
2021-05-18 15:49:22 -04:00
David Steele
dc0e6645cd Move includes after typedef in info/manifest.h.
This allows the Manifest type to be used in headers that need it, e.g. command/backup/common.h.
2021-05-18 09:08:06 -04:00
David Steele
9af033194a
Add automatic GCS authentication for GCE instances.
When running on a GCE instance the authentication token can be pulled directly from the instance metadata. This is configured with repo-gcs-key-type=auto.

In a separate commit (26fefa6), move the code that parses the token response into a separate function, storageGcsAuthToken(), since it is now needed by two key types. This drastically improves the readability of the main commit.
2021-05-17 14:55:50 -04:00
David Steele
0152075e6b Remove default VM for test.pl --coverage-only option.
When running outside of our standard Vagrantfile the default will not be set correctly, so require the user to set it.

In any case, this option is primarily useful for reporting so note that in the command line help.
2021-05-17 11:35:22 -04:00
David Steele
384f247077 Add config/parse tests for config/env partial options.
Partial option matching is valid on the command line but should never be used in config files or environment options.
2021-05-17 09:33:36 -04:00
David Steele
ae7f0af202 Move PostgreSQL version interface test functions to a test harness.
Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.

Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
David Steele
18cc02238b Remove overzealous cast.
There does not seem to be a good reason to have an explicit cast for such a small number, so remove it.

Leave PG_CONTROL_SIZE alone since it will be removed in an upcoming commit.
2021-05-17 07:15:13 -04:00
David Steele
9235c62c6b Revert ignoring catalog version when identifying a PostgreSQL version.
927d9adb changed the way CATALOG_VERSION_NO is used to identify PostgreSQL versions since PG_CONTROL_VERSION is generally bumped with each release. The goal was to make the beta/rc period less painful because any CATALOG_VERSION_NO bump renders pgBackRest inoperative.

This worked, but in fact we'd rather be stricter about which CATALOG_VERSION_NO we accept when identifying a version of PostgreSQL. It is not just about identifying a major version, but making sure the build contains all the functions and catalogs we expect to make pgBackRest work correctly. It is better to reject early dev/beta/rc builds that may not work.

Since 927d9adb was relatively recent the chance that this stricter checking will cause a problem seems minimal, so revert to checking CATALOG_VERSION_NO for every PostgreSQL version.

Leave in place the code that pulls CATALOG_VERSION_NO from pg_control rather than the internal constant since the plan is still to allow catalog versions to "float" during the PostgreSQL beta/rc phase, which will be the subject of a future commit.
2021-05-17 07:12:46 -04:00
David Steele
f45e76fa2e
Fix issues with leftover spool files from a prior restore.
If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue.

Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process.

Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
David Steele
34dd6636b8 Add config/parse tests for options and option args with spaces.
If an option name has a space at the beginning then it will be considered an invalid command, but a space at the end is an invalid option. Add tests for these conditions.

Spaces in option arguments should be preserved, so add a test to be sure this is true.
2021-05-13 17:01:21 -04:00
David Steele
2b8d2daca1 Add config/parse tests for partial options.
Add coverage for partial options, i.e. an option that is partially specified but only prefix matches with a single valid option.
2021-05-13 16:54:25 -04:00
David Steele
add0f9b77d Fix an ungrammatical error message in config parsing. 2021-05-13 12:37:59 -04:00
David Steele
0825428fef Add config/parse test where the option/value are not in the same arg.
All the tests in this module use --option=value syntax so add one test with --option value syntax for coverage.
2021-05-13 12:35:11 -04:00
David Steele
26fefa6aee Refactor GCS token response parsing into a separate function.
This function is also useful for the upcoming auto authentication. Refactoring in a separate commit to make the feature commit more readable.
2021-05-13 06:59:57 -04:00
David Steele
7b71604def Use existing variable for GCS test server port.
Also fix incorrect separator comment.
2021-05-12 13:21:59 -04:00
David Steele
5464ac83d1
Convert option values in commands to StringId.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.

For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
2021-05-11 17:24:30 -04:00
David Steele
87ba2ca253 Change CentOS 7 documentation test to CentOS 8.
The CentOS 7 documentation test relies on PostgreSQL 9.5 which has been removed from the yum.p.o repository package. Switch the test to CentOS 8 to fix the immediate issue, but a decision on the PostgreSQL 9.5 documentation will need to be made before the next release.
2021-05-11 16:54:42 -04:00
David Steele
2bfebff2ec Add instructions to install lcov for MacOS. 2021-05-04 15:51:46 -04:00
David Steele
54074a5697 Exclude Dockerfile from code count. 2021-05-04 07:52:35 -04:00
David Steele
baddec1e9a Basic multi-architecture support for test containers.
The tests worked fine on multiple architectures, but would only run "bare metal", i.e. tests that required containers could not be run.

Enable basic multi-architecture support by allowing containers to be built using whatever architecture the host supports. Also allow cached containers to be defined for multiple architectures in container.yaml.

Add a Dockerfile which can be used as a container for other containers to provide a consistent development environment.

The primary goal is to allow development on Mac M1 but other architectures should find these improvements useful.
2021-05-03 16:31:27 -04:00
David Steele
87df6d7a58
Convert BackupType enum to StringId.
Allows removal of backupType()/backupTypeStr() and improves debug logging of the enum.

Move BackupType enum and string constants to info/infoBackup.h so they are available to more modules. Also convert InfoBackup to use BackupType instead of a String.
2021-05-03 12:15:39 -04:00
David Steele
568dc0ba0c Add new instructions for PostgreSQL news and include a sample.
It is no longer possible to pull news source from the PostgreSQL website so add a sample in the doc directory. Update the release instructions to reflect this change.

Also note that it is no longer necessary to post separately to pgsql-announce.
2021-05-03 09:34:22 -04:00
David Steele
fb7ddce807 Convert InfoPgType enum to StringId.
Improves debug logging by outputting archive/backup instead of 0/1.
2021-04-28 13:21:24 -04:00
David Steele
bd0081fec8 Update IoClient/IoSession to use StringIds.
Using StringId for the client/session type removes String constants and some awkward referencing/dereferencing needed to use a String constant in the interface.

Converting IoSessionRole to StringId removes a conditional in ioSessionToLog() and improves debug logging by outputting client/server instead of 0/1.
2021-04-28 12:37:22 -04:00
David Steele
0ec91f61c6 Convert ArchivePushFileIoType enum to StringId.
Improves debug logging by outputting open/write/close instead of 0/1/2.
2021-04-28 12:23:37 -04:00
David Steele
8394577c6a Convert ArchiveMode enum to StringId.
Improves debug logging by outputting get/push instead of 0/1.
2021-04-28 12:05:01 -04:00
David Steele
7dd01897fd Convert ProtocolStorageType enum to StringId.
Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
David Steele
bd68ed63ba Convert ProtocolParallelJobState enum to StringId.
Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
David Steele
85fc3da4c3
Update CipherType/CipherMode to StringId.
As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
Cynthia Shang
c3b15fc3bd Fix comments where 'output' was misspelled as 'ouput'. 2021-04-28 10:58:45 -04:00
David Steele
066fbcf268 Refactor String, Buffer, and Variant types with inline getters.
Extend the pattern introduced in 79a2d02c to the String, Buffer, and Variant types.
2021-04-27 15:25:10 -04:00
David Steele
1edcfde93e
Add cfgOptionDisplay()/cfgOptionIdxDisplay().
Centralize the formatting of the configuration value for display to the user or passing on a command line.

For the new functions, if the value was set by the user via the command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and saves having to format it into something reasonable, especially for time and size option types.

Note that cfgOptTypeHash and cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole.

This also fixes a bug in config/load.c where time values where not being formatted correctly in an error message.
2021-04-27 12:12:43 -04:00
David Steele
95fb002b85 Preserve unused YAML fix for older Debian/Ubuntu distributions.
This fix was not used since backports worked for older Debians and the old affected Ubuntus are EOL.

Still, seems worth preserving just in case it comes up elsewhere.
2021-04-27 12:04:50 -04:00
David Steele
6cc521b6b2
Update storage module to use StringIds.
Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.

Also add new config functions and harness config functions to support StringIds.
2021-04-23 13:19:47 -04:00
David Steele
aa72c19a83
Do not write files atomically or sync paths during backup copy.
There is no need to write the file atomically (e.g. via a temp file on Posix) because checksums are tested on resume after a failed backup. The path does not need be synced for each file because all paths are synced at the end of the backup.

This functionality was not lost during the migration -- it never existed in the Perl code, though these settings are used in restore. See 59f1353 where backupFile() was migrated to C.
2021-04-23 12:33:25 -04:00
David Steele
aaa15b9709
Add help for all internal options valid for default roles.
Fix the segfault when getting help for an internal option is requested by adding help for all internal options that are valid for a default command role.

Also print warnings about internal options in code rather than putting in each command/option description.
2021-04-23 11:46:03 -04:00
David Steele
2ad497ea4c Remove documentation about enum truncation and add tests.
The enum truncation observed was due to the value getting passed via a protocol function which silently narrowed the enum.

Even so, add some tests to ensure tested platforms support 64-bit enums.
2021-04-23 08:04:03 -04:00
David Steele
bcc925b740 Replace misused kvAdd() with kvPut().
Although kvAdd() works like kvPut() on the first call, kvPut() is more efficient when a key has a single value.

Update the comment to clarify that kvAdd() is seldom required.
2021-04-22 20:04:27 -04:00
David Steele
a2f02f95ec Remove redundant CompressType conversions.
These seem to be the result of overzealous copy/paste.

The conversions do not even serve as a verification of locally available compression types since compressTypeEnum() and compressTypeStr() do not check that.
2021-04-22 19:28:11 -04:00
David Steele
468aa79ea8 Remove auto-generated option String constants.
These are rarely used and end up using more space than they save since most of them are never referenced.

Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
David Steele
6a39c51f8f Test pull requests in Cirrus CI.
Both FreedBSD and MacOS use clang as their default compiler and clang catches some errors that gcc does not. Specifically, wide integers being assigned to short integers resulting in possible truncations.
2021-04-22 18:18:34 -04:00
David Steele
06fa18582a
Free no longer needed remotes so they do not timeout during restore.
The remotes are no longer needed in the main process after the manifest is loaded. If the restore is long enough the connection will timeout and WARN at the end of the restore. This is harmless for the restore but distracting for the user.

To prevent this, free the remotes once they are no longer needed.
2021-04-22 12:23:08 -04:00
David Steele
45f83558ea
Fix help when a valid option is invalid for the specified command.
Getting help for a valid option that was invalid for the command would segfault.

Add a check to ensure the option is valid for the command's default role.
2021-04-22 11:48:04 -04:00
David Steele
72dac5b10b Use option constants in warnings.
In once case the constant replaces a formatted statement, which is more efficient, and in the other case plain text, which is more maintainable.
2021-04-22 08:15:36 -04:00
David Steele
20a018a8ae Fix incorrect test title. 2021-04-21 18:27:18 -04:00
David Steele
c5b446d406 Fix indentation. 2021-04-21 10:09:07 -04:00