1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

153 Commits

Author SHA1 Message Date
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
bf0691576a Allow delta option to be specified in the pgBackRest configuration file.
This option was previously allowed on the command-line only for no particular reason that we could determine.

Being able to specify it in the config file seems like a good idea and won't change current usage.

Contributed by Cynthia Shang.
2018-09-18 13:29:05 -04:00
David Steele
d41570c37a Improve log file names for remote processes started by locals.
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes.  Not only was the local command used for the naming of log files but the process id was not pass through.  This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.

Instead, pass the real command and process id to the remote.  This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
2018-08-31 11:31:13 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
David Steele
77dca5b968 Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added. 2018-08-24 19:31:45 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
238e5dae3c Allow resolve array from cd5df357 to autosize to detect short arrays in testing. 2018-08-14 17:17:14 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
cb975a5330 Remove redundant lines from embedded Perl by combining blank lines. 2018-07-28 08:43:58 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Cynthia Shang
721432f6ae Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024.
Contributed by Cynthia Shang.
2018-04-25 15:46:05 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
ab26c765d4 Don't copy CFGDEF_NAME_ALT or CFGDEF_INHERIT when processing config option inheritance. 2018-04-18 18:57:45 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
741acfd779 Move test definitions to test/define.yaml.
The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 10:19:24 -04:00
David Steele
2c1687721e Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected.
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.

Reported by Clinton Adams.
2018-03-19 13:08:42 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
f047cd0dfd Improve code generation performance. 2018-03-09 23:00:20 -05:00
David Steele
5e0ce3e996 Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate.
Reported by Brad Nicholson.
2018-03-06 17:12:16 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
Cynthia Shang
4352407777 Option rename cleanup based on review.
* Remove --reset tags from v1 options.
* Use constants for repo prefix.
* Specify CFGDEF_INDEX_PG in option structure rather than adding in code.
* Fix error message references to "backup host".

Contributed by Cynthia Shang.
2018-02-19 10:12:51 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp().
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
Cynthia Shang
00e9aca872 Fixed issue where --reset-* was not passing the default value to Perl.
Contributed by Cynthia Shang.
2018-02-13 15:58:14 -05:00
David Steele
ea296372d4 Add allow range to int/float options that did not already have one. 2018-02-09 15:42:11 -05:00
David Steele
7c58fe7451 Use 64-bit integers for config options.
In particular, archive-queue-max requires a greater range than is provided by a 32-bit integer.
2018-02-09 13:54:33 -05:00
David Steele
1659598cfe Create a master list of errors in build/error.yaml.
The C and Perl errors lists are created automatically by Build.pm so they stay up to date.
2018-02-08 16:11:47 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
d35757c05f Fix missing log-level-stderr option for stanza-* commands. 2018-02-05 15:19:28 -05:00
David Steele
db21b7a360 Cleanup usage of internal options.
Apply internal to options that need to be read to determine locality but should not appear in the help.
2018-02-05 11:50:09 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
3dd882a79e Add internal flag to config definition. 2018-01-23 12:49:04 -05:00
David Steele
bffc6c49b3 Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work. 2018-01-16 13:23:08 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated.
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore.
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00
David Steele
5eb5d94d65 Allow internal options that do not show up in the documentation.
Used for test options initially but other use cases are on the horizon.
2017-12-05 14:46:52 -05:00
David Steele
8d4674031e More config parsing in C in preparation for all config parsing in C. 2017-12-05 10:09:07 -05:00
David Steele
4f1725dc36 Replace cfgCommandTotal()/cfgOptionTotal() functions with constants.
The constants are applicable in more cases and allow the compiler to optimize certain loops more efficiently.
2017-12-05 09:45:51 -05:00
David Steele
915ae5662a Implement version command in C. 2017-11-28 21:44:05 -05:00
David Steele
2ed426a70b Remove deprecated archive-max-mb option. 2017-11-27 20:43:44 -05:00
David Steele
7cf955425e The C library is now required.
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
55f68ef013 Remove configurable option hints.
db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.
2017-10-30 10:58:04 -04:00
David Steele
1ef27ec8c2 Add list type for options.
The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.
2017-10-30 10:50:35 -04:00
David Steele
8efcc38304 Improvements in C codebase:
* Update C naming conventions.
* Use int datatype wherever possible.
* Better separation of C source from Perl interface.
2017-09-30 10:44:03 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00