1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3079 Commits

Author SHA1 Message Date
David Steele
87996558d2
Replace double type with time in config module.
The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.

Also remove Variant doubles since they are no longer used.

Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
David Steele
a137c6baaa Update install-sh, config.guess, and config.sub to latest versions. 2020-12-07 11:26:34 -05:00
David Steele
ab0500789e Begin v2.32 development. 2020-12-07 11:13:45 -05:00
David Steele
e116b535e6 v2.31: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)

Features:

* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
* Add pg-database option. (Reviewed by Cynthia Shang.)

Improvements:

* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)

Documentation Improvements:

* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
2020-12-07 09:55:00 -05:00
David Steele
31becf05b7 Add RHEL/CentOS 8 documentation.
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.

Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
2020-12-04 10:59:57 -05:00
David Steele
b0ea337965 Add pg-database option.
In some rare cases there is no postgres database so this option may be used to specify an alternate database.
2020-12-02 22:42:50 -05:00
David Steele
1913ecee63 Fix incorrect usage of TEST_RESULT_DOUBLE().
These work fine but it's better to use the correct macro.
2020-12-02 22:42:05 -05:00
David Steele
d4211d3aaf Add retries to PostgreSQL sleep when starting a backup.
Inaccuracies in sleep time or clock skew might make a single sleep insufficient to reach the next second.

Add a few retries to make the process more reliable but still avoid an infinite loop if something is seriously wrong.
2020-12-02 22:41:14 -05:00
David Steele
ec9f23d31f Remove CentOS 6 from tests and documentation.
CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.

Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.
2020-12-02 16:23:05 -05:00
David Steele
ffc50719d9 Improve error message when PQgetCancel() returns NULL.
There is not a lot to be done in this case since it looks like PostgreSQL disconnected while the query was running, but at least improve the error message and remove the assert, which indicates a coding error.
2020-12-01 15:15:35 -05:00
David Steele
d1d25c710d Remove extraneous storageRepo() calls.
These calls are not required since cipher info is passed explicitly. They are probably a copy-pasto from some past time when one of these functions required it.
2020-11-30 18:03:24 -05:00
Stefan Fercot
5488de8b6a
Report page checksum errors in info command text output.
This feature currently only works for text output. JSON output is planned for the future.
2020-11-25 12:14:03 -05:00
Cynthia Shang
3ed7b93b90 Conform retry in lockAcquireFile() to the common retry pattern. 2020-11-24 09:40:44 -05:00
David Steele
117f03eba1 Prepare configuration module for multi-repository support.
Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).

Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.

Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.

Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.

Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.

Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.

Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
David Steele
7fda83b31e
Allow multiple remote locks from the same main process.
Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.

The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
David Steele
77198dce97 Update Vagrant box version. 2020-11-22 16:26:30 -05:00
Stefan Fercot
191b8ec18b
Create standby.signal only on PostgreSQL 12 when restore type is standby.
When restore type standby is provided, the recovery.signal isn't needed and may lead to some confusion (see #1236).

Lately, when using pg_basebackup --write-recovery-conf, only the standby.signal file is created. This change would then align with that behaviour.
2020-11-19 16:57:19 -05:00
Cynthia Shang
9aacd3c54a
Move retrieval of archiveResult before while loop in verifyArchive().
The result structure for the archive id being processed only needs to be retrieved once so moving it outside of the WAL path list processing loop is more efficient.
2020-11-18 18:19:49 -05:00
David Steele
62d9f23961 Rename hrnCfgEnvId*() to hrnCfgEnvKey*().
The hrnCfg*() functions were renamed before commit but these got missed.
2020-11-17 10:48:40 -05:00
David Steele
dd708e6d27 Fix override of user option reset.
If a user reset an option such as pg-default on the command-line then an override in the code would not take effect.

Ignore a reset when the code explicitly sets an option to prevent this.
2020-11-09 16:37:05 -05:00
David Steele
6acd76b60c Fix test title in storage/remote test. 2020-11-09 16:28:53 -05:00
David Steele
63c12d2541 Fix spacing and typos in backup/manifest modules. 2020-11-09 16:26:43 -05:00
David Steele
3f7a66fffc Remove obsolete call to storageRepo() in archivePushProtocol().
This call to storageRepo() was used to fetch cipher options from a remote to determine if a repo cipher was enabled.

Now the main process does this work and passes the cipher options directly to the local so there is no need to pre-load the repo storage here.
2020-11-09 16:19:37 -05:00
David Steele
cbb9b8fd2b Check archive push queue limit before checking repository.
If the push queue limit has been exceeded then nothing will be pushed to the repo so there is no point in checking it. Worse, a failure in the check would cause drop not to run and potentially fill up the disk, exactly the case this feature was designed to prevent.

The async version already checks the push queue limit before checking the repository so now both versions have the same behavior.
2020-11-09 16:16:41 -05:00
David Steele
8cf47f82e4 Log warnings in archive-push async log.
These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen.

This also improves coverage by requiring a warning during async processing to have a test case, which has been added.
2020-11-09 16:10:59 -05:00
David Steele
d5d1ec6f6f Use a constant to check restore target action.
Checking the default here was fragile. If the default were to change the code would break.

This also removes the only dependency on cfgOptionDefault() outside of the help command.
2020-11-04 11:09:05 -05:00
Cynthia Shang
021a34a55b Fix typos. 2020-10-29 17:16:15 -04:00
David Steele
f87a117b71 Spell out three exclamation points in git-history.cache.
Three exclamation points are used by convention as a marker for code that needs attention before it can be committed to integration.

If the markers are in this file they come up in every search.
2020-10-28 08:59:43 -04:00
David Steele
d25e965c47 Use harnessCfgLoad() in config/load unit tests.
The tests were originally written by loading values directly into the configuration before the parser was available.

Update to use harnessCfgLoad() to simplify the tests and make them compatible with upcoming config changes.
2020-10-27 13:29:58 -04:00
David Steele
d91e6014bd Fix comment. 2020-10-27 13:18:25 -04:00
Stefan Fercot
abe9d90c89
Improve info command output when a stanza is specified but missing.
Return a path missing error when a stanza is specified for the info command but the stanza does not exist in the repository.

Previously [] was returned, which is still the case if no stanza is specified and the repository does not exist.
2020-10-27 08:34:18 -04:00
David Steele
770b65de80
Improve performance of large file lists in backup/restore commands.
lstRemoveIdx(list, 0) resulted in the entire list being moved down to the first position which could take a long time for big lists. This is a common pattern in backup/restore when processing file queues.

Instead simply move the list pointer up when first item is removed. Then on insert check if there is space at the beginning when there is no longer space at the end and do the move then. This way if a list is built and then drained without any new inserts then no move is required.
2020-10-26 12:18:45 -04:00
David Steele
d452e9cc38
Use zero-based indexes when referring to option indexes.
There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa.

Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user.

Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
Cynthia Shang
ae35c4f029
Remove unused FUNCTION_LOG_VERIFY_WAL_RANGE* defines.
The defines for FUNCTION_LOG_VERIFY_WAL_RANGE* are not used in the current verify.c and are currently not planned in the continuing development of the verify command, so they are dead code and are therefore being removed.
2020-10-26 07:41:08 -04:00
David Steele
ea49151746 Replace misuse of bufSize() with bufUsed().
bufSize() should only be used whem checking the total size of the buffer, not how much of it is currently used.

In these cases bufUsed() and bufSize() are returning the same value but benign-looking code changes could break this assumption.
2020-10-24 13:18:02 -04:00
David Steele
80a770ed2e Clarify intended usage of harnessCfgLoadRaw(). 2020-10-24 11:16:06 -04:00
David Steele
76cfd8ca70
Allow [, #, and space as the first character in database names.
iniLoad() was trimming lines which meant that a leading space would not pass checksum validation when a manifest was reloaded. Remove the trims since files we write should never contain extraneous spaces. This further diverges the format for the functions that read conf files (e.g. pgbackrest.conf) and those that read info (e.g. manifest) files.

While we are at it also allow [ and # as initial characters. # was reserved for comments but we never put comments into info files. [ denotes a section but we can get around this by never allowing arrays as values in info files, so if a line ends in ] it must be a section. This is currently the case but enforce it by adding an assert to info/info.c.
2020-10-24 11:07:07 -04:00
David Steele
c573ef2814 Add indexed option > 1 to config/exec unit test. 2020-10-20 16:14:03 -04:00
David Steele
176cf0bf60 Use harnessCfgLoadRaw() in command/command and common/exit unit tests.
The tests were originally written by loading values directly into the configuration before the parser was available.

Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes.

Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.
2020-10-20 14:54:28 -04:00
David Steele
156b7d48cc Get target-action default from cfgOptionDefault() in restore command.
cfgDefOptionDefault() worked but the default is available without having to peek into config definitions.
2020-10-20 12:39:23 -04:00
David Steele
41789d70d1 Remove cfgOptionId() and replace it with cfgParseOption().
cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.

cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().

Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.
2020-10-20 11:24:26 -04:00
David Steele
6414ae9707 Remove ConfigDefineCommand enum.
This has always been equivalent to the ConfigCommand enum so it just adds complexity.

It was created for symmetry with ConfigDefineOption, which will also be removed soon.
2020-10-19 18:17:47 -04:00
David Steele
ba29e24684 Fix repo/pg-local option being overwritten.
This option would get overwritten when the option values were moved to index 1.

Set the repo/pg-local option after options are moved to fix this.
2020-10-19 17:17:47 -04:00
David Steele
2d38d2fc82 Reset additional options in real/all integration test.
Currently indexes above 1 do not have dependencies checked, so this doesn't error.

In a future commit we will enable those checks and this will error if it is not fixed.
2020-10-19 17:06:52 -04:00
David Steele
996387ea93 Use harnessCfgLoadRaw() in command/help unit test.
This correctly set defaults from cfgLoadUpdateOption(), e.g. repo-host-cmd.
2020-10-19 16:27:52 -04:00
David Steele
303b9b17e7 Remove extraneous calls to cfgOptionValid() in unit tests. 2020-10-19 14:19:28 -04:00
David Steele
7d069a2b91 Remove indexed option constants.
These constants don't scale well as the index total is increased for an option.

The core code rarely uses these options and they are easily replaced with cfgOptionName().

The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
Stefan Fercot
86275c4f85
Expire history files.
WAL timeline history files were not being expired because they were small and generally not very plentiful.

However, in some cases large numbers of history files may be generated so it makes sense to remove useless history files to keep things tidy.

The history file for the oldest retained timeline is kept for debugging purposes even though it is not used for recovery.
2020-10-16 07:42:03 -04:00
David Steele
4b8dfd34c0 Assert when buffer used is greater than size limit.
099281c5 reduced used but this could lead to surprising behavior so assert instead.

Call bufUsedSet() before bufLimitSet() to avoid the assertion.
2020-10-08 17:09:21 -04:00
David Steele
401e4f1df4
Optimize small reads in IoRead.
Instead of using memmove() to manage the internal output buffer for every small read, track the current buffer position and only move data when the small read cannot be satisfied and more data is needed.
2020-10-08 16:09:38 -04:00