1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

67 Commits

Author SHA1 Message Date
David Steele
fe263e87b1 Allow path-style URIs in S3 driver.
Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.

Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.
2020-01-12 11:31:06 -07:00
David Steele
7de5ce23ad Add internal remote-type option.
This option was overloaded on the general type option but it makes sense to split this out since the meaning is pretty different.

Rename the values to conform to current standards, i.e. pg and repo, now that the Perl code won't care anymore.
2020-01-08 18:59:02 -07:00
David Steele
74c3842595 Remove errant tabs and fix spacing. 2019-12-19 16:25:46 -05:00
Mike Palmiotto
7dfea6b717 Remove errant tab. 2019-12-19 16:20:11 -05:00
Mike Palmiotto
dc1e7ca22d Add pg-user option.
Specifies the database user name when connecting to PostgreSQL.

If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior.
2019-12-19 11:26:38 -05:00
David Steele
2f7e139534 Error if option prefix and index total are not both defined.
They are both required for indexed options.
2019-12-17 22:35:30 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
c933f12f9c Remove obsolete --perl-option option.
This option was used when Perl was executed instead of being embedded.

It has been obsolete for a long time so remove it.
2019-12-10 13:28:15 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
5f03471445 Remove --force option from stanza-create documentation.
This should have been removed when the support for the option was removed in c7333190.

The option cannot be removed entirely because we don't want to error in the case where --force was specified but the stanza is valid.
2019-11-19 15:56:13 -05:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
15d04ca19c Add recursion and json output to the ls command.
These features finally make the ls command practical.

Currently the JSON contains only name, type, and size.  We may add more fields in the future, but these seem like the minimum needed to be useful.
2019-09-12 16:29:50 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
5cbe2dee85 Add compress-level-network option to ls command.
039e515a added a dependency on compress-level-network but the ls command did not get updated.
2019-06-25 07:36:16 -04:00
David Steele
6ff3325c77 Enforce requiring repo-cipher-pass at config parse time.
This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.

Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
2019-06-05 11:43:17 -04:00
David Steele
404284b90f Add internal flag for commands.
Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.

Update help for ls now that it is exposed.
2019-05-28 12:18:05 -04:00
David Steele
20e5b92f36 Add ls command.
Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.

This command is internal-only so the interface may change at any time without notice.
2019-05-28 10:03:48 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
David Steele
15531151d7 Add --c option to request a C remote.
The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated.

Instead, set the --c option when a C remote is required.  This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
David Steele
c72a94cfec Increase process-max limit to 999.
Recent hardware has outstripped the old default so increase it to something that should last for a while.

Suggested by Rakshitha-BR.
2019-04-07 18:20:03 -04:00
David Steele
251dbede8f Add locking capability to the remote command.
When a repository server is configured, commands that modify the repository acquire a remote lock as well as a local lock for extra protection against multiple writers.

Instead of the custom logic used in Perl, make remote locking part of the command configuration.

This also means that the C remote needs the stanza since it is used to construct the lock name.  We may need to revisit this at a later date.
2019-03-27 21:14:06 +00:00
David Steele
9382283586 Fix issues when a path option is / terminated.
This condition was not being properly checked for in the C code and it caused problems in the info command, at the very least.

Instead of applying a local fix, introduce a new path option type that will rigorously check the format of any incoming paths.

Reported by Marc Cousin.
2019-03-14 13:48:33 +04:00
David Steele
b8ebea6b1c Add separate archive-push-async command.
This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-03-14 13:38:55 +04:00
David Steele
60fe5b7365 Error when parameters are passed to a command that does not accept parameters.
This behavior allowed a command like this to run without error:

pgbackrest backup --stanza=db full

Even though it actually performed an incremental backup in most circumstances because the `full` parameter was ignored.

Instead, output an error and exit.

Suggested by Jason O'Donnell.
2019-03-09 11:03:47 +02:00
David Steele
9367cc461c Migrate local command to C.
The C local is only used for C commands in the main process.

Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
David Steele
73be64ce49 Add separate archive-get-async command.
This command was previously forked off from the archive-get command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-02-20 15:52:07 +02:00
David Steele
64b97fd7ca Correct archive-get-queue-max to be size type.
This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.

This is not a breaking change because currently configured integer values will be read as bytes.

Reported by Ronan Dunklau.
2018-11-30 10:55:29 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
6680130c6f Require S3 key options except for local/remote commands.
S3 key options (repo1-s3-key/repo1-s3-key-secret) were not required which meant that users got an ugly assertion when they were missing rather than a tidy configuration error.

Only the local/remote commands need them to be optional.  This is because local/remote commands get all their options from the command line but secrets cannot be passed on the command line.  Instead, secrets are passed to the local/remote commands via the protocol for any operation that needs them.

The configuration system allows required to be set per command so use that to improve the error messages while not breaking the local/remote commands.
2018-11-20 19:24:53 -05:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
bf0691576a Allow delta option to be specified in the pgBackRest configuration file.
This option was previously allowed on the command-line only for no particular reason that we could determine.

Being able to specify it in the config file seems like a good idea and won't change current usage.

Contributed by Cynthia Shang.
2018-09-18 13:29:05 -04:00
David Steele
d41570c37a Improve log file names for remote processes started by locals.
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes.  Not only was the local command used for the naming of log files but the process id was not pass through.  This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.

Instead, pass the real command and process id to the remote.  This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
2018-08-31 11:31:13 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Cynthia Shang
721432f6ae Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024.
Contributed by Cynthia Shang.
2018-04-25 15:46:05 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
ab26c765d4 Don't copy CFGDEF_NAME_ALT or CFGDEF_INHERIT when processing config option inheritance. 2018-04-18 18:57:45 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
2c1687721e Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected.
The max value for a local/remote process is now error since there's no reason for these processes to emit warnings.

Reported by Clinton Adams.
2018-03-19 13:08:42 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
5e0ce3e996 Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate.
Reported by Brad Nicholson.
2018-03-06 17:12:16 -05:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00