1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3177 Commits

Author SHA1 Message Date
David Steele
7f263dceb7 Fix inverted cfgOption*() calls.
These currently work because repoIdx is always 0.
2021-01-15 08:30:56 -05:00
David Steele
298cc4d5e5 Remove non-conforming periods and reformat some comments. 2021-01-14 10:39:25 -05:00
David Steele
c2c702c09d Add co7 package to support llvm.
This is required for new package versions.

Also remove the obsolete 9.2 package and update the supported versions list.
2021-01-13 17:32:42 -05:00
David Steele
a57e0be395 Add duplicate WAL test to synchronous archive-get. 2021-01-13 13:01:40 -05:00
David Steele
22fd223fc3 Improve logging in archive-get command.
Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though).

Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful.

Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
David Steele
c6aaf66e9d Add FUNCTION_LOG_RETURN_STRUCT() and update where appropriate.
The FUNCTION_LOG_RETURN() macro requires logging macros (e.g. FUNCTION_LOG_*_TYPE and FUNCTION_LOG_*_FORMAT) when returning a struct but these macros don't deliver much value since they only output the name of the struct rather than the contents. A copy of the struct is also made during this operation, which is wasteful.

FUNCTION_LOG_RETURN_STRUCT() does not make a copy of the struct and does not require any logging macros. Returned structures are logged as "struct" but this could be made more accurate using __typeof in the future.

Structures as parameters are not addressed here and work as before, i.e. they require logging macros.
2021-01-13 07:49:47 -05:00
David Steele
b21ed97982 Check for missing files in queueNeed().
Missing files would indicate that another process is running on the same spool path, which would be a very bad thing.

This check doesn't cost any additional I/O so it seems like a good idea.
2021-01-12 18:47:19 -05:00
David Steele
aae6f06bcf Add tests for the archive-get command.
Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
David Steele
aeee83044d
Fix resume after partial delete of backup by prior resume.
If files other than backup.manifest.copy were left in a backup path by a prior resume then the next resume would skip the backup rather than removing it. Since the backup path still existed, it would be found during backup label generation and cause an error if it appeared to be later than the new backup label. This occurred if the skipped backup was full.

The error was only likely on object stores such as S3 because of the order of file deletion. Posix file systems delete from the bottom up because directories containing files cannot be deleted. Object stores do not have directories so files are deleted in whatever order they are provided by the list command. However, the issue can be reproduced on a Posix file system by manually deleting backup.manifest.copy from a resumable backup path.

Fix the issue by removing the resumable backup if it has no manifest files. Also add a new warning message for this condition.

Note that this issue could be resolved by running expire or a new full backup.
2021-01-12 12:38:32 -05:00
David Steele
96fd678662
Add job-retry and job-retry-interval options.
These options specify the number of local worker job retries and the retry interval after one immediate retry.

There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.

Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.

These options are currently internal.
2021-01-11 15:15:25 -05:00
David Steele
8b86d5ea7a Restore storageRepo() call in archiveGetProtocol().
This call was removed by d1d25c71, which worked for archivePushProtocol() and verifyProtocol() since the encryption options are passed from the main process.

archiveGetProtocol() still retrieves these options in the local process so the repo storage must be loaded first.
2021-01-11 11:34:03 -05:00
David Steele
f35d69c1c7 Refactor common/archiveGet unit test.
The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor.

Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
David Steele
8567b7f733 Make archive-get locality error generate a global.error file.
Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
David Steele
7d3d6ecbe1 Add storage test macros for common operations.
The macros should make it much easier to write complex tests, especially when compression and encryption are involved.

Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
David Steele
97c260bb56 Add TEST_RESULT_STRLST_*() unit test macros.
This avoids the need for strLstJoin() when testing lists.

Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
David Steele
1ef36b4032 Add clarifications to unit test string diff messages. 2021-01-08 10:54:17 -05:00
David Steele
17e29eb1bd Improve formatting of unit test titles.
Add separation and some visual cues to help identify the start of a test.

Also add a counter which can be used to search for a specific test, which is useful if there is a lot of debug output to search through.
2021-01-08 10:45:26 -05:00
David Steele
dc0284412b Add test paths for pg, repo, and spool.
These may be used as standard locations for these paths in tests.
2021-01-08 10:35:26 -05:00
David Steele
6bdbcdc7fe Create TEST_PATH define in test.c.
This allows paths to be constructed without using sprintf(), which makes the tests simpler and faster.
2021-01-08 10:32:57 -05:00
David Steele
ca9c1707f0 Remove forks from command/archive-get test.
These were required to deal with the legacy Perl code being unable to load new options between tests.

The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
David Steele
6e7a3eb383 Remove archive-timeout from test in mock/archive.
No timeout is expected here but the small timeout prevents errors from being thrown.

This is not a bug since the error would be thrown on the next archive-get call but it does make the tests harder to debug when there is an error.

It is not clear why there was a timeout here at all. It is likely cruft from a prior test or a copy/paste error.
2021-01-05 18:11:28 -05:00
Cynthia Shang
656e711296
Remove duplicate tests from command/infoTest.
Tests that are duplicated are being removed from the info command unit tests. Specifically tests where the only thing different was whether a lock was held or not which affects only the status display. Removing these tests will reduce churn in the upcoming multi-repo support.
2021-01-05 15:14:16 -05:00
David Steele
a7491182dc Set build-max to 2 for Github actions.
Github runners only provide 2 virtual CPUs so the default build-max of 4 is not ideal.
2021-01-04 09:18:17 -05:00
David Steele
9345fe4cb2 Update LICENSE.txt and PostgreSQL copyright for 2020. 2021-01-03 13:56:37 -05:00
David Steele
af17bc4b6d Update test to work with different collations.
The data returned by the protocol has not been sorted yet so it is vulnerable to differences in collation.

Multiple records are not needed for this test so limit it to one path to solve this issue.
2020-12-31 15:34:07 -05:00
David Steele
0e1612cda1 Remove explicit command lists where they equal the default.
This reduces noise in the file and new commands will automatically get these options.
2020-12-31 12:29:11 -05:00
David Steele
108038292c Audit options valid for expire command. 2020-12-31 12:13:20 -05:00
David Steele
0acfcb669e Audit options valid for start/stop commands. 2020-12-31 11:10:48 -05:00
David Steele
82a75268c1 Remove option exclusions now handled by command role validity.
These options were explicitly excluded because it was possible for them to be mangled by SSH if they contained spaces.

They are now excluded by command role validity rules.
2020-12-31 10:22:09 -05:00
David Steele
09fdde359c Limit pg option validity and make it command-line only.
The pg option only has one current usage, to let the backup local know which pg index it should copy files from.

There are other possible uses for this option, but they need thought, tests, and documentation.
2020-12-31 10:08:58 -05:00
David Steele
951cfa9e90 Remove repo option.
This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be.

The option will be added back when multi-repo functionality is committed.
2020-12-31 08:12:35 -05:00
David Steele
e3762945c2 Call storageRemoteInfoParse() in prior context to avoid dups.
This results in fewer data duplications and makes the code less fragile since new data add in storageRemoteInfoParse() does not need to be added to an additional list for duplication.
2020-12-30 17:33:33 -05:00
Cynthia Shang
cc90163233
Add empty archive array to info command JSON when stanza is missing.
There is an inconsistency when the JSON is output for the case when a stanza is requested and it does not exist in the repo. This was the only case where the archive array was not added to the JSON. Adding it will simplify the upcoming multi-repo support code.

Also, a redundant test was removed rather than updating it for this case.
2020-12-30 16:17:56 -05:00
David Steele
9bf7dbf6a2 Do not pass pg-local/repo-local to a remote process.
This was a hack to prevent the remote from loading host settings, which is now handled by option validity for command roles.

These options are still useful so don't remove them, but do leave them internal for now.
2020-12-30 16:03:49 -05:00
David Steele
141466875f Remove redundant command list in repo-s3-key option.
Use the repo-type command list as similar repo options do.
2020-12-30 10:51:26 -05:00
David Steele
abb8ebe58b
Limit option validity by command role.
Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.

The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
David Steele
26e81785d9 Update unit tests for stricter option validity.
Some tests used options in contexts that are currently valid but are not correct usage, i.e. usage of internal options for the default role.

Update these tests in advance of the option validity becoming stricter.
2020-12-28 15:22:21 -05:00
David Steele
23f5712d02
Allow option validity to be determined by command role.
Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them.

Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command.

This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.
2020-12-28 09:43:23 -05:00
David Steele
715fa7a2f1 Fix indentation error preventing pull requests from being tested. 2020-12-28 08:51:22 -05:00
David Steele
9e9e7c4a0d Move all parse-related rules to parse module.
Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl.

Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement.

Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
David Steele
f520ecc89a Move help data from define.auto.c/config.auto.c to a pack.
The help data can be represented more compactly in a pack and this separates data needed for help from data needed for parsing, freeing each to have a more appropriate representation.
2020-12-16 15:59:36 -05:00
David Steele
996de0a3e6 Remove cfgCmdNone from CFG_COMMAND_TOTAL.
cfgCmdNone is used to indicate a missing or invalid command so should not be used in the total used for command process.
2020-12-16 11:33:51 -05:00
David Steele
558ff1e555 Always advance id when pack field is NULL.
This was done in the internal versions but not the user-facing function. That meant the field had to be explicitly read after determining it was NULL, which is wasteful.

Since there is only one behavior now, remove pckReadDefaultNull() and move the logic to pckReadNullInternal().
2020-12-16 09:59:48 -05:00
David Steele
39963f6aa5 Remove cfgDefOptionIndexTotal().
This function was only used in one place, which was better served by cfgOptionGroupIdxTotal().
2020-12-14 14:37:23 -05:00
David Steele
7f66ba5762 Remove deprecated option from command/archive-push unit test. 2020-12-10 17:58:59 -05:00
David Steele
7429aca8fa Remove redundant distro specification. 2020-12-09 17:26:52 -05:00
David Steele
d01669aa58 Move most tests to Github Actions.
Testing on Travis-CI has been getting slower (from ~18 minutes to 3-6 hours) and the travis-ci.org service will be terminated at the end of the year. Moving to travis-ci.com is an option but the quotas are too low for our purposes.

Instead use Github Actions, which does not currently have quotas, and runs our current tests with just a few tweaks.

This still leaves multi-architecture tests on Travis-CI but we may be able to run those and stay within the new quotas.

Also fix a minor bug in restoreTest.c exposed by Github Actions using a different name for the user and group.
2020-12-09 15:19:01 -05:00
David Steele
8361a97482
Add pack type.
The pack type is an architecture-independent format for serializing data compactly, inspired by ProtocolBuffers and Avro.

Also add ioReadSmall(), which is optimized for small binary reads, similar to ioReadLineParam().
2020-12-09 12:05:14 -05:00
David Steele
87996558d2
Replace double type with time in config module.
The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.

Also remove Variant doubles since they are no longer used.

Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
David Steele
a137c6baaa Update install-sh, config.guess, and config.sub to latest versions. 2020-12-07 11:26:34 -05:00