1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

3702 Commits

Author SHA1 Message Date
David Steele
2b8d2daca1 Add config/parse tests for partial options.
Add coverage for partial options, i.e. an option that is partially specified but only prefix matches with a single valid option.
2021-05-13 16:54:25 -04:00
David Steele
add0f9b77d Fix an ungrammatical error message in config parsing. 2021-05-13 12:37:59 -04:00
David Steele
0825428fef Add config/parse test where the option/value are not in the same arg.
All the tests in this module use --option=value syntax so add one test with --option value syntax for coverage.
2021-05-13 12:35:11 -04:00
David Steele
26fefa6aee Refactor GCS token response parsing into a separate function.
This function is also useful for the upcoming auto authentication. Refactoring in a separate commit to make the feature commit more readable.
2021-05-13 06:59:57 -04:00
David Steele
7b71604def Use existing variable for GCS test server port.
Also fix incorrect separator comment.
2021-05-12 13:21:59 -04:00
David Steele
5464ac83d1
Convert option values in commands to StringId.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.

For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
2021-05-11 17:24:30 -04:00
David Steele
87ba2ca253 Change CentOS 7 documentation test to CentOS 8.
The CentOS 7 documentation test relies on PostgreSQL 9.5 which has been removed from the yum.p.o repository package. Switch the test to CentOS 8 to fix the immediate issue, but a decision on the PostgreSQL 9.5 documentation will need to be made before the next release.
2021-05-11 16:54:42 -04:00
David Steele
2bfebff2ec Add instructions to install lcov for MacOS. 2021-05-04 15:51:46 -04:00
David Steele
54074a5697 Exclude Dockerfile from code count. 2021-05-04 07:52:35 -04:00
David Steele
baddec1e9a Basic multi-architecture support for test containers.
The tests worked fine on multiple architectures, but would only run "bare metal", i.e. tests that required containers could not be run.

Enable basic multi-architecture support by allowing containers to be built using whatever architecture the host supports. Also allow cached containers to be defined for multiple architectures in container.yaml.

Add a Dockerfile which can be used as a container for other containers to provide a consistent development environment.

The primary goal is to allow development on Mac M1 but other architectures should find these improvements useful.
2021-05-03 16:31:27 -04:00
David Steele
87df6d7a58
Convert BackupType enum to StringId.
Allows removal of backupType()/backupTypeStr() and improves debug logging of the enum.

Move BackupType enum and string constants to info/infoBackup.h so they are available to more modules. Also convert InfoBackup to use BackupType instead of a String.
2021-05-03 12:15:39 -04:00
David Steele
568dc0ba0c Add new instructions for PostgreSQL news and include a sample.
It is no longer possible to pull news source from the PostgreSQL website so add a sample in the doc directory. Update the release instructions to reflect this change.

Also note that it is no longer necessary to post separately to pgsql-announce.
2021-05-03 09:34:22 -04:00
David Steele
fb7ddce807 Convert InfoPgType enum to StringId.
Improves debug logging by outputting archive/backup instead of 0/1.
2021-04-28 13:21:24 -04:00
David Steele
bd0081fec8 Update IoClient/IoSession to use StringIds.
Using StringId for the client/session type removes String constants and some awkward referencing/dereferencing needed to use a String constant in the interface.

Converting IoSessionRole to StringId removes a conditional in ioSessionToLog() and improves debug logging by outputting client/server instead of 0/1.
2021-04-28 12:37:22 -04:00
David Steele
0ec91f61c6 Convert ArchivePushFileIoType enum to StringId.
Improves debug logging by outputting open/write/close instead of 0/1/2.
2021-04-28 12:23:37 -04:00
David Steele
8394577c6a Convert ArchiveMode enum to StringId.
Improves debug logging by outputting get/push instead of 0/1.
2021-04-28 12:05:01 -04:00
David Steele
7dd01897fd Convert ProtocolStorageType enum to StringId.
Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
David Steele
bd68ed63ba Convert ProtocolParallelJobState enum to StringId.
Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
David Steele
85fc3da4c3
Update CipherType/CipherMode to StringId.
As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
Cynthia Shang
c3b15fc3bd Fix comments where 'output' was misspelled as 'ouput'. 2021-04-28 10:58:45 -04:00
David Steele
066fbcf268 Refactor String, Buffer, and Variant types with inline getters.
Extend the pattern introduced in 79a2d02c to the String, Buffer, and Variant types.
2021-04-27 15:25:10 -04:00
David Steele
1edcfde93e
Add cfgOptionDisplay()/cfgOptionIdxDisplay().
Centralize the formatting of the configuration value for display to the user or passing on a command line.

For the new functions, if the value was set by the user via the command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and saves having to format it into something reasonable, especially for time and size option types.

Note that cfgOptTypeHash and cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole.

This also fixes a bug in config/load.c where time values where not being formatted correctly in an error message.
2021-04-27 12:12:43 -04:00
David Steele
95fb002b85 Preserve unused YAML fix for older Debian/Ubuntu distributions.
This fix was not used since backports worked for older Debians and the old affected Ubuntus are EOL.

Still, seems worth preserving just in case it comes up elsewhere.
2021-04-27 12:04:50 -04:00
David Steele
6cc521b6b2
Update storage module to use StringIds.
Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.

Also add new config functions and harness config functions to support StringIds.
2021-04-23 13:19:47 -04:00
David Steele
aa72c19a83
Do not write files atomically or sync paths during backup copy.
There is no need to write the file atomically (e.g. via a temp file on Posix) because checksums are tested on resume after a failed backup. The path does not need be synced for each file because all paths are synced at the end of the backup.

This functionality was not lost during the migration -- it never existed in the Perl code, though these settings are used in restore. See 59f1353 where backupFile() was migrated to C.
2021-04-23 12:33:25 -04:00
David Steele
aaa15b9709
Add help for all internal options valid for default roles.
Fix the segfault when getting help for an internal option is requested by adding help for all internal options that are valid for a default command role.

Also print warnings about internal options in code rather than putting in each command/option description.
2021-04-23 11:46:03 -04:00
David Steele
2ad497ea4c Remove documentation about enum truncation and add tests.
The enum truncation observed was due to the value getting passed via a protocol function which silently narrowed the enum.

Even so, add some tests to ensure tested platforms support 64-bit enums.
2021-04-23 08:04:03 -04:00
David Steele
bcc925b740 Replace misused kvAdd() with kvPut().
Although kvAdd() works like kvPut() on the first call, kvPut() is more efficient when a key has a single value.

Update the comment to clarify that kvAdd() is seldom required.
2021-04-22 20:04:27 -04:00
David Steele
a2f02f95ec Remove redundant CompressType conversions.
These seem to be the result of overzealous copy/paste.

The conversions do not even serve as a verification of locally available compression types since compressTypeEnum() and compressTypeStr() do not check that.
2021-04-22 19:28:11 -04:00
David Steele
468aa79ea8 Remove auto-generated option String constants.
These are rarely used and end up using more space than they save since most of them are never referenced.

Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
David Steele
6a39c51f8f Test pull requests in Cirrus CI.
Both FreedBSD and MacOS use clang as their default compiler and clang catches some errors that gcc does not. Specifically, wide integers being assigned to short integers resulting in possible truncations.
2021-04-22 18:18:34 -04:00
David Steele
06fa18582a
Free no longer needed remotes so they do not timeout during restore.
The remotes are no longer needed in the main process after the manifest is loaded. If the restore is long enough the connection will timeout and WARN at the end of the restore. This is harmless for the restore but distracting for the user.

To prevent this, free the remotes once they are no longer needed.
2021-04-22 12:23:08 -04:00
David Steele
45f83558ea
Fix help when a valid option is invalid for the specified command.
Getting help for a valid option that was invalid for the command would segfault.

Add a check to ensure the option is valid for the command's default role.
2021-04-22 11:48:04 -04:00
David Steele
72dac5b10b Use option constants in warnings.
In once case the constant replaces a formatted statement, which is more efficient, and in the other case plain text, which is more maintainable.
2021-04-22 08:15:36 -04:00
David Steele
20a018a8ae Fix incorrect test title. 2021-04-21 18:27:18 -04:00
David Steele
c5b446d406 Fix indentation. 2021-04-21 10:09:07 -04:00
David Steele
fd69357302 Add const to inline functions where appropriate.
This lets the compiler know that these variables are not modified which should lead to better optimization.

Smart compilers should be able to figure this out on their own, but marking parameters const is still good for documentation.
2021-04-20 18:43:16 -04:00
David Steele
ed0d48f52c Add StringId type.
It is often useful to represent identifiers as strings when they cannot easily be represented as an enum/integer, e.g. because they are distributed among a number of unrelated modules or need to be passed to remote processes. Strings are also more helpful in debugging since they can be recognized without cross-referencing the source. However, strings are awkward to work with in C since they cannot be directly used in switch statements leading to less efficient if-else structures.

A StringId encodes a short string into an integer so it can be used in switch statements but may also be readily converted back into a string for debugging purposes. StringIds may also be suitable for matching user input providing the strings are short enough.

This patch includes a sample of StringId usage by converting protocol commands to StringIds. There are many other possible use cases. To list a few:

* All "types" in storage, filters. IO , etc. These types are primarily for identification and debugging so they fit well with this model.

* MemContext names would work well as StringIds since these are entirely for debugging.

* Option values could be represented as StringIds which would mean we could remove the functions that convert strings to enums, e.g. CipherType.

* There are a number of places where enums need to be converted back to strings for logging/debugging purposes. An example is protocolParallelJobToConstZ. If ProtocolParallelJobState were defined as:

typedef enum
{
    protocolParallelJobStatePending = STRID5("pend", ...),
    protocolParallelJobStateRunning = STRID5("run", ...),
    protocolParallelJobStateDone = STRID5("done", ...),
} ProtocolParallelJobState;

then protocolParallelJobToConstZ() could be replaced with strIdToZ(). This also applies to many enums that we don't covert to strings for logging, such as CipherMode.

As an example of usage, convert all protocol commands from strings to StringIds.
2021-04-20 15:22:42 -04:00
Stefan Fercot
292f836f12
Add db-exclude option.
Restore excluding the specified databases. Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.

When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres).
2021-04-19 15:01:00 -04:00
David Steele
a75b413ac2 Add storageDriver() inlines missed in b715c70b. 2021-04-19 14:22:36 -04:00
David Steele
d859fe8c4d Update to remove files after archive-get test completes.
This cleans up the spool path for the next test. Currently there isn't one but there will be.
2021-04-14 15:48:04 -04:00
David Steele
040ad71f8c Remove lockClear().
This function has not been used since the switch to the fork/exec model.

lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
David Steele
c2d4a0286e Define DEBUG in build.auto.c.
Both NDEBUG and DEBUG were used in the code, which was a bit confusing.

Define DEBUG in build.auto.c so it is available in all C and header files and stop using NDEBUG. This is preferable to using NDEBUG everywhere since there are multiple DEBUG* defines, e.g. DEBUG_COVERAGE.

Note that NDEBUG is still required since it is used by the C libraries.
2021-04-13 18:06:07 -04:00
David Steele
c1aae434ca Update Cirrus CI FreeBSD release to 12.2. 2021-04-13 17:03:28 -04:00
David Steele
a0e24d492f Add THIS_PUB() macro to simplify inline object accessors.
In combination with the thisPub() function, this macro simplifies accessing the public part of a private object struct.

thisPub() asserts this != NULL so the caller does not need to do it.
2021-04-13 16:58:02 -04:00
David Steele
9fec4ce98c Refactor remaining common/io modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the remaining common/io modules.
2021-04-13 14:37:02 -04:00
Isaacwhyuenac
5bf160643b Clarify that repo-s3-role is not an ARN. 2021-04-13 14:02:20 -04:00
David Steele
2c9c59170f Add bash to Cirrus CI FreeBSD package install.
It appears bash was removed from the base image so install explicitly.
2021-04-13 12:38:17 -04:00
David Steele
8844ced384 Refactor common/io/filter module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/io/filter module.
2021-04-12 16:05:40 -04:00
David Steele
3b9bed9518 Refactor common/type/keyValue module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/type/keyValue module.
2021-04-09 17:29:14 -04:00