1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

152 Commits

Author SHA1 Message Date
David Steele
96fd678662
Add job-retry and job-retry-interval options.
These options specify the number of local worker job retries and the retry interval after one immediate retry.

There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.

Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.

These options are currently internal.
2021-01-11 15:15:25 -05:00
David Steele
0e1612cda1 Remove explicit command lists where they equal the default.
This reduces noise in the file and new commands will automatically get these options.
2020-12-31 12:29:11 -05:00
David Steele
108038292c Audit options valid for expire command. 2020-12-31 12:13:20 -05:00
David Steele
0acfcb669e Audit options valid for start/stop commands. 2020-12-31 11:10:48 -05:00
David Steele
09fdde359c Limit pg option validity and make it command-line only.
The pg option only has one current usage, to let the backup local know which pg index it should copy files from.

There are other possible uses for this option, but they need thought, tests, and documentation.
2020-12-31 10:08:58 -05:00
David Steele
951cfa9e90 Remove repo option.
This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be.

The option will be added back when multi-repo functionality is committed.
2020-12-31 08:12:35 -05:00
David Steele
9bf7dbf6a2 Do not pass pg-local/repo-local to a remote process.
This was a hack to prevent the remote from loading host settings, which is now handled by option validity for command roles.

These options are still useful so don't remove them, but do leave them internal for now.
2020-12-30 16:03:49 -05:00
David Steele
141466875f Remove redundant command list in repo-s3-key option.
Use the repo-type command list as similar repo options do.
2020-12-30 10:51:26 -05:00
David Steele
abb8ebe58b
Limit option validity by command role.
Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.

The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
David Steele
23f5712d02
Allow option validity to be determined by command role.
Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them.

Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command.

This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.
2020-12-28 09:43:23 -05:00
David Steele
9e9e7c4a0d Move all parse-related rules to parse module.
Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl.

Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement.

Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
David Steele
f520ecc89a Move help data from define.auto.c/config.auto.c to a pack.
The help data can be represented more compactly in a pack and this separates data needed for help from data needed for parsing, freeing each to have a more appropriate representation.
2020-12-16 15:59:36 -05:00
David Steele
996de0a3e6 Remove cfgCmdNone from CFG_COMMAND_TOTAL.
cfgCmdNone is used to indicate a missing or invalid command so should not be used in the total used for command process.
2020-12-16 11:33:51 -05:00
David Steele
39963f6aa5 Remove cfgDefOptionIndexTotal().
This function was only used in one place, which was better served by cfgOptionGroupIdxTotal().
2020-12-14 14:37:23 -05:00
David Steele
87996558d2
Replace double type with time in config module.
The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.

Also remove Variant doubles since they are no longer used.

Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
David Steele
b0ea337965 Add pg-database option.
In some rare cases there is no postgres database so this option may be used to specify an alternate database.
2020-12-02 22:42:50 -05:00
David Steele
117f03eba1 Prepare configuration module for multi-repository support.
Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).

Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.

Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.

Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.

Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.

Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.

Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
David Steele
7fda83b31e
Allow multiple remote locks from the same main process.
Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.

The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
David Steele
41789d70d1 Remove cfgOptionId() and replace it with cfgParseOption().
cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.

cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().

Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.
2020-10-20 11:24:26 -04:00
David Steele
6414ae9707 Remove ConfigDefineCommand enum.
This has always been equivalent to the ConfigCommand enum so it just adds complexity.

It was created for symmetry with ConfigDefineOption, which will also be removed soon.
2020-10-19 18:17:47 -04:00
David Steele
7d069a2b91 Remove indexed option constants.
These constants don't scale well as the index total is increased for an option.

The core code rarely uses these options and they are easily replaced with cfgOptionName().

The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
David Steele
e0f09687e4
Add option groups.
Group related options together so operations (e.g. valid, test, index total) can be performed on all options in the group.

Previously, options at the top of the hierarchy of the related options were used to do these tests. This was prone to error as option relationships changed and it was not always clear which option (or options) should be used.
2020-10-08 10:52:19 -04:00
David Steele
98f6a1cffd Remove unused cfgDefOptionPrefix() function and data.
This function was made obsolete when Perl was removed in f0ef73db.
2020-10-07 12:15:50 -04:00
David Steele
9377d05072
Add repo-azure-endpoint option.
This option allows alternate endpoints (e.g. Azure Government) to be configured.
2020-10-06 17:15:48 -04:00
Cynthia Shang
ad79932ba5
Add internal verify command.
Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet).

The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
David Steele
8c2960fab3
Add archive-mode option to disable archiving on restore.
When restoring a cluster that will be promoted but is not intended to be the new primary, it is important to disable archiving to avoid polluting the repository with useless WAL. This option makes disabling archiving a bit easier.
2020-08-25 15:05:41 -04:00
David Steele
851f2e814e
Automatically retrieve temporary S3 credentials on AWS instances.
Automatically retrieve the role and temporary credentials for S3 when the AWS instance is associated with an IAM role. Credentials are automatically updated when they are <= 5 minutes from expiring.

Basic configuration is to set repo1-s3-key-type=auto. repo1-s3-role can be used to set a specific role, otherwise it will be retrieved automatically.
2020-08-25 10:38:49 -04:00
Stefan Fercot
d3dd32a031
Add expire-auto option.
This allows automatic expiration after a successful backup to be disabled.
2020-07-14 08:12:25 -04:00
David Steele
2f7823c627
Add shared access signature (SAS) authorization for Azure.
A shared access signature (SAS) provides granular, delegated access to resources in a storage account. This is often preferable to using a shared key which provides more access and is a greater security risk if compromised.
2020-07-09 14:46:48 -04:00
David Steele
3f4371d7a2 Azure support for repository storage.
Azure and Azure-compatible object stores can now be used for repository storage.

Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
David Steele
3d74ec1190
Use PostgreSQL instead of postmaster where appropriate.
Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.
2020-06-17 15:14:59 -04:00
David Steele
da4f15663b Improve error when pg1-path option missing for archive-get command.
The assert thrown was not as descriptive as a proper option missing error.
2020-06-10 11:41:08 -04:00
David Steele
20d8c76b6c
Ignore pg-host* and repo-host* options for the remote command.
The purpose of the remote command is to get access to local resources, so a remote should never start another remote. However, this could happen if there were host settings on the remote host, which ended badly with lock errors, loops, etc.

Add pg-local and repo-local options to indicate that the resource is local even if there are host settings.

Note that for the time being these options are internal and not intended for general usage. However, this is likely the direction needed to allow for more symmetric and manageable configurations.
2020-05-22 13:51:26 -04:00
David Steele
ea9147e2e0 Reduce buffer-size default to 1MiB.
The prior default was determined by benchmarking the Perl code prior to the 1.0 release. In general buffer allocation was more expensive in Perl so large buffers gave the best performance. This was due to multiple buffer allocations for each filter in an IO operation.

The C code allocates fixed buffers for each IO operation so the cost for buffer allocation is lower than Perl. That being the case it made sense to benchmark the C code to determine the optimal buffer default.

The performance/storage tests were used to measure the performance of a variety of filters. 1GiB of data was processed by each filter 10 times and the results of the tests were averaged.

While most buffer sizes gave similar performance, 1MiB appeared to perform the best overall. Of course, different architectures are likely to yield different results but this seems like a sensible default. The buffer-size option may still need to be manually configured to give optimal results.

Raw test data for reference:

4MB buffer (prior default)

copy time 1807ms, avg time 180ms, avg throughput: 5942MB/s
md5 time 14200ms, avg time 1420ms, avg throughput: 756MB/s
sha1 time 11431ms, avg time 1143ms, avg throughput: 939MB/s
sha256 time 23463ms, avg time 2346ms, avg throughput: 457MB/s
gzip -6 time 381199ms, avg time 38119ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

1MB buffer (new default)

copy time 1760ms, avg time 176ms, avg throughput: 6100MB/s
md5 time 13739ms, avg time 1373ms, avg throughput: 781MB/s
sha1 time 11025ms, avg time 1102ms, avg throughput: 973MB/s
sha256 time 22539ms, avg time 2253ms, avg throughput: 476MB/s
gzip -6 time 372995ms, avg time 37299ms, avg throughput: 28MB/s
lz4 -1 time 15118ms, avg time 1511ms, avg throughput: 710MB/s

512K buffer

copy time 1782ms, avg time 178ms, avg throughput: 6025MB/s
md5 time 13724ms, avg time 1372ms, avg throughput: 782MB/s
sha1 time 10959ms, avg time 1095ms, avg throughput: 979MB/s
sha256 time 22982ms, avg time 2298ms, avg throughput: 467MB/s
gzip -6 time 378120ms, avg time 37812ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

256K buffer

copy time 1805ms, avg time 180ms, avg throughput: 5948MB/s
md5 time 13706ms, avg time 1370ms, avg throughput: 783MB/s
sha1 time 11074ms, avg time 1107ms, avg throughput: 969MB/s
sha256 time 22588ms, avg time 2258ms, avg throughput: 475MB/s
gzip -6 time 372645ms, avg time 37264ms, avg throughput: 28MB/s
lz4 -1 time 16346ms, avg time 1634ms, avg throughput: 656MB/s
2020-05-19 16:58:49 -04:00
Cynthia Shang
cdebfb09e0
Add time-based retention for full backups.
The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days.

The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 15:25:03 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
Cynthia Shang
1c1a710460 Add --set option to the expire command.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 14:00:36 -04:00
David Steele
6f41ce1182 Add expire to commands that support io-timeout.
The command list was not complete. Copy the command list from buffer-size to make sure all commands are included.
2020-04-20 09:45:08 -04:00
David Steele
21a1cf9a2f Add sck-block option.
This option puts sockets into blocking mode.

It is only intended for internal use and is not included in the test matrix.
2020-04-17 09:55:40 -04:00
David Steele
5d25e508ae Add io-timeout option.
Timeout used for connections and read/write operations.

Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte.
2020-04-17 09:18:52 -04:00
David Steele
f19c59ab38 Remove unused exports from pgBackRestBuild::Config::Data.
These were once used heavily in the Perl code but now only a few of them are used in the documentation.

Remove the unused exports as there is generally no need create them for new commands/options.
2020-04-01 18:03:16 -04:00
David Steele
789e364e6b Rename tcp-keep-alive option to sck-keep-alive.
This is really a socket option so the new name is clearer.

Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.
2020-04-01 15:44:51 -04:00
David Steele
5c6fb88bef TCP keep-alive options are configurable.
Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.

Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.

Also move configuration of TCP options before connect() as PostgreSQL does.
2020-03-31 18:13:11 -04:00
Cynthia Shang
048a36ea39 Order commands alphabetically in definition of set option. 2020-03-26 13:04:59 -04:00
Cynthia Shang
2fa69af8da Add --dry-run option to the expire command.
Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.
2020-03-16 13:56:52 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00