1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-06-14 23:44:58 +02:00
Commit Graph

96 Commits

Author SHA1 Message Date
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
78b1e05b7e Remove unused Perl module. 2019-12-07 17:39:25 -05:00
0933aeb1ad Remove unused reference to use pgBackRest::Backup::File. 2019-11-29 10:56:13 -05:00
158e439689 Remove obsolete Perl archive code.
This should have been removed in a1c13a50 but was missed.
2019-11-26 17:16:45 -05:00
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
7d8068f27b Don't decode manifest data when it is generated on a remote.
Decoding a manifest from the JSON provided by C to the hash required by Perl is an expensive process.  If manifest() was called on a remote it was being decoded into a hash and then immediately re-encoded into JSON for transmission over the protocol layer.

Instead, provide a function for the remote to get the raw JSON which can be transmitted as is and decoded in the calling process instead.

This makes remote manifest calls as fast as they were before 2.16, but local calls must still pay the decoding penalty and are therefore slower.  This will continue to be true until the Perl storage interface is retired at the end of the C migration.

Note that for reasonable numbers of tables there is no detectable difference.  The case in question involved 250K tables with a 10 minute decode time (which was being doubled) on a fast workstation.
2019-09-03 12:30:45 -04:00
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
92e04ea9f4 Remove per-stanza repo cache clear during testing.
This was not being used and is not supported by the equivalent C code.
2019-06-04 10:34:19 -04:00
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
1bc84c6474 The local command for restore is implemented entirely in C.
This is just the part of restore run by the local helper processes, not the entire command.

Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 17:07:37 -04:00
cb00030ee3 Remove dead code missed in 1b486847.
This commit removed all Perl references to spool storage but some stuff was left behind.
2019-05-08 18:58:07 -04:00
1b48684713 The archive-push command is implemented entirely in C.
This new implementation should behave exactly like the old Perl code with the exception of updated log messages.

Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
db4b447be8 The archive-get command is implemented entirely in C.
This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages.

Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
d245f8eb42 The info command is implemented entirely in C.
The C info code has already been committed but this commit wires it into main.

Also remove the info Perl code and tests since they are no longer called.
2019-01-21 13:51:45 +02:00
b4146b6bff Update Perl repo rules to work when stanza is not specified.
The C storage object strives to use rules whenever possible instead of generating absolute paths.  This change helps the C and Perl storage work together via the protocol layer.
2019-01-16 18:45:19 +02:00
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
b6b2c915b2 Allow hashSize() to run on remote storage.
Apparently we never needed to run this function remotely.

It will be needed by the backup checksum delta feature, so implement it now.

Contributed by Cynthia Shang.
2018-09-18 11:39:48 -04:00
d41570c37a Improve log file names for remote processes started by locals.
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes.  Not only was the local command used for the naming of log files but the process id was not pass through.  This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.

Instead, pass the real command and process id to the remote.  This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
2018-08-31 11:31:13 -04:00
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
74d6398ad2 The pgbackrest executable is now a C binary instead of Perl.
This allows certain time-critical commands (like async archive-push) to run more quickly.
2017-11-26 18:43:51 -05:00
ac54aac448 Fixed an issue passing parameters to remotes.
When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed.

Reported by Uspen.
2017-11-21 13:17:44 -05:00
c77fc1fa61 Refactor protocol param generation into a new function.
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
062e714307 Disable gzip filter when --compress-level-network=0.
The filter was used with compress level set to 0 which added overhead without any benefit.
2017-11-21 09:31:15 -05:00
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore.
Reported by Craig A. James.
2017-11-14 14:48:44 -05:00
b03c26968a Repository encryption support.
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
d989cf8ac2 Replace dynamically built class hierarchies in I/O layer with fixed parent() calls. 2017-10-22 19:07:17 -04:00
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
44c7c0475a Improve protocol error handling.
In particular, "stop" errors are no longer reported as "unexpected".
2017-10-21 14:11:33 -04:00
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
6e29f616b3 Harden protocol handshake to handle race conditions. 2017-06-15 15:12:28 -04:00
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00