1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

90 Commits

Author SHA1 Message Date
David Steele
c933f12f9c Remove obsolete --perl-option option.
This option was used when Perl was executed instead of being embedded.

It has been obsolete for a long time so remove it.
2019-12-10 13:28:15 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
8dfe0e48e2 Use more general error code when tablespace linked into PGDATA.
The specific error code was not that useful since we also test the error message which contains details of the link error.
2019-12-02 10:49:25 -05:00
David Steele
5f03471445 Remove --force option from stanza-create documentation.
This should have been removed when the support for the option was removed in c7333190.

The option cannot be removed entirely because we don't want to error in the case where --force was specified but the stanza is valid.
2019-11-19 15:56:13 -05:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
15d04ca19c Add recursion and json output to the ls command.
These features finally make the ls command practical.

Currently the JSON contains only name, type, and size.  We may add more fields in the future, but these seem like the minimum needed to be useful.
2019-09-12 16:29:50 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
Josh Soref
d1cc22f886 Fix typo in CFGDEFDATA_OPTION_OPTIONAL_COMMAND_OVERRIDE macro name.
Contributed by Josh Soref.
2019-08-26 11:25:14 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
5cbe2dee85 Add compress-level-network option to ls command.
039e515a added a dependency on compress-level-network but the ls command did not get updated.
2019-06-25 07:36:16 -04:00
David Steele
0a96a2895d Add storage layer for tests and documentation.
The tests and documentation have been using the core storage layer but soon that will depend entirely on the C library, creating a bootstrap problem (i.e. the storage layer will be needed to build the C library).

Create a simplified Posix storage layer to be used by documentation and the parts of the test code that build and execute the actual tests.  The actual tests will still use the core storage driver so they can interact with any type of storage.
2019-06-17 09:16:44 -04:00
David Steele
6ff3325c77 Enforce requiring repo-cipher-pass at config parse time.
This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.

Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
2019-06-05 11:43:17 -04:00
David Steele
404284b90f Add internal flag for commands.
Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.

Update help for ls now that it is exposed.
2019-05-28 12:18:05 -04:00
David Steele
20e5b92f36 Add ls command.
Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.

This command is internal-only so the interface may change at any time without notice.
2019-05-28 10:03:48 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
David Steele
15531151d7 Add --c option to request a C remote.
The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated.

Instead, set the --c option when a C remote is required.  This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
David Steele
c5157c0334 Automatically generate constants for command and option names.
These constants are easier than using cfgOptionName() and cfgCommandName() and lead to cleaner code and simpler to construct messages.

String versions are provided. Eventually all the strings will be used in the config structures, but for now they are useful to avoid wrapping with strNew().
2019-04-12 09:03:34 -04:00
David Steele
c72a94cfec Increase process-max limit to 999.
Recent hardware has outstripped the old default so increase it to something that should last for a while.

Suggested by Rakshitha-BR.
2019-04-07 18:20:03 -04:00
David Steele
251dbede8f Add locking capability to the remote command.
When a repository server is configured, commands that modify the repository acquire a remote lock as well as a local lock for extra protection against multiple writers.

Instead of the custom logic used in Perl, make remote locking part of the command configuration.

This also means that the C remote needs the stanza since it is used to construct the lock name.  We may need to revisit this at a later date.
2019-03-27 21:14:06 +00:00
David Steele
9382283586 Fix issues when a path option is / terminated.
This condition was not being properly checked for in the C code and it caused problems in the info command, at the very least.

Instead of applying a local fix, introduce a new path option type that will rigorously check the format of any incoming paths.

Reported by Marc Cousin.
2019-03-14 13:48:33 +04:00
David Steele
b8ebea6b1c Add separate archive-push-async command.
This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-03-14 13:38:55 +04:00
David Steele
60fe5b7365 Error when parameters are passed to a command that does not accept parameters.
This behavior allowed a command like this to run without error:

pgbackrest backup --stanza=db full

Even though it actually performed an incremental backup in most circumstances because the `full` parameter was ignored.

Instead, output an error and exit.

Suggested by Jason O'Donnell.
2019-03-09 11:03:47 +02:00
David Steele
9367cc461c Migrate local command to C.
The C local is only used for C commands in the main process.

Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
David Steele
73be64ce49 Add separate archive-get-async command.
This command was previously forked off from the archive-get command which required a bit of artificial option and log manipulation.

A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-02-20 15:52:07 +02:00
David Steele
9cac403f61 Add Exec object.
Executes a child process and allows the calling process to communicate with it using read/write io.

This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose
execution.
2019-01-18 11:45:40 +02:00
David Steele
64b97fd7ca Correct archive-get-queue-max to be size type.
This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.

This is not a breaking change because currently configured integer values will be read as bytes.

Reported by Ronan Dunklau.
2018-11-30 10:55:29 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
6680130c6f Require S3 key options except for local/remote commands.
S3 key options (repo1-s3-key/repo1-s3-key-secret) were not required which meant that users got an ugly assertion when they were missing rather than a tidy configuration error.

Only the local/remote commands need them to be optional.  This is because local/remote commands get all their options from the command line but secrets cannot be passed on the command line.  Instead, secrets are passed to the local/remote commands via the protocol for any operation that needs them.

The configuration system allows required to be set per command so use that to improve the error messages while not breaking the local/remote commands.
2018-11-20 19:24:53 -05:00
David Steele
b0659278cc Add ServiceError for errors from a service that can be retried.
An example is HTTP 5xx errors which should mostly be retried.
2018-11-16 17:22:22 -05:00
David Steele
72ea47bfb3 Add KernelError to report miscellaneous kernel errors. 2018-11-11 18:07:56 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
bf0691576a Allow delta option to be specified in the pgBackRest configuration file.
This option was previously allowed on the command-line only for no particular reason that we could determine.

Being able to specify it in the config file seems like a good idea and won't change current usage.

Contributed by Cynthia Shang.
2018-09-18 13:29:05 -04:00
David Steele
d41570c37a Improve log file names for remote processes started by locals.
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes.  Not only was the local command used for the naming of log files but the process id was not pass through.  This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.

Instead, pass the real command and process id to the remote.  This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
2018-08-31 11:31:13 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
David Steele
77dca5b968 Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added. 2018-08-24 19:31:45 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
238e5dae3c Allow resolve array from cd5df357 to autosize to detect short arrays in testing. 2018-08-14 17:17:14 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
7993f1a966 Add basic C JSON parser. 2018-08-09 08:06:23 -04:00
David Steele
cb975a5330 Remove redundant lines from embedded Perl by combining blank lines. 2018-07-28 08:43:58 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
8c6e2bdbc7 Add storageInfo() and track size in read objects. 2018-04-29 11:02:21 -04:00