1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

126 Commits

Author SHA1 Message Date
David Steele
1db9e3b144 Remove *MP() macros variants.
Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.

-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
Cynthia Shang
db1dc4f275 Remove pretty-printing from jsonFromKv() and jsonFromVar().
Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.
2019-10-11 13:03:52 -04:00
David Steele
1049632873 Add user module for managing system users/groups.
Centralize the management of users and groups.

Also update Posix storage driver where users/groups were already in use.
2019-09-08 20:11:51 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
2862f480cd Add special file type to storageInfo().
There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.
2019-08-23 07:24:25 -04:00
David Steele
0da3352d59 Add cvtZToMode() replacement missed in 969e5817. 2019-08-17 17:52:00 -04:00
David Steele
d8ca0e5c5b Add Perl interface to C PgQuery object.
This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.
2019-07-25 17:05:39 -04:00
David Steele
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
David Steele
38ba458616 Add IoSink filter.
Discard all data passed to the filter.  Useful for calculating size/checksum on a remote system when no data needs to be returned.

Update ioReadDrain() to automatically use the IoSink filter.
2019-07-18 08:42:42 -04:00
David Steele
fc21013522 Fix scoping violations exposed by optimizations in gcc 9.
gcc < 9 makes all compound literals function scope, even though the C spec requires them to be invalid outside the current scope.  Since the compiler and valgrind were not enforcing this we had a few violations which caused problems in gcc >= 9.

Even though we are not quite ready to support gcc 9 officially, fix the scoping violations that currently exist in the codebase.

Reported by chrlange, Ned T. Crigler.
2019-07-05 16:25:28 -04:00
David Steele
4bffa0c5bb Add test function to create the S3 bucket instead of using aws cli.
Eventually the idea is to remove the dependency on aws cli since Python is a big install.
2019-06-26 15:02:30 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
David Steele
3a5fd7b28a Fix typo in macro name. 2019-05-30 11:19:59 -04:00
David Steele
a474ba54c5 Refactoring path support in the storage module.
Not all storage types support paths as a physical thing that must be created/destroyed.  Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object.

Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense.

Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.
2019-05-26 12:41:15 -04:00
David Steele
32ca27a20b Simplify storage object names.
Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent.

Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
David Steele
8c712d89eb Improve type safety of interfaces and drivers.
The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations.  Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been.

Use void pointers instead so that no casts are required.  Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer.

Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type.  This cuts down on accessor methods and the vast majority of those functions were not being used.  Move functions that are still required to .intern.h.

Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
David Steele
027c263871 Add configure script for improved multi-platform support.
Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile.

Each C file must include "build.auto.conf" before all other includes and defines.  This is enforced by test.pl for includes, but it won't detect incorrect define ordering.

Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
David Steele
2ef5ad70a2 Move crypto module to common/crypto.
It makes sense for the crypto code to be in common since it is not pgBackRest-specific.

Also combine the crypto tests into a single module.
2019-03-10 13:27:30 +02:00
David Steele
95597be81e Move compress module to common/compress.
It makes sense for the compression code to be in common since it is not pgBackRest-specific.
2019-03-10 13:11:20 +02:00
David Steele
1f66bda02e Fix non-compliant JSON for options passed from C to Perl.
We have been using a hacked-up JSON generator to pass options from C to Perl since the C binary was introduced.  This generator was not very compliant which led to issues with \n, ", etc. inside strings.

We have a fully-compliant JSON generator now so use that instead.

Reported by Leo Khomenko.
2019-02-22 12:02:26 +02:00
David Steele
1de22cac2b Rename common/io/handle module to common/io/handleWrite.
ioHandleWriteOneStr() will become a helper function for the IoHandleWrite object.
2019-01-06 14:37:39 +02:00
David Steele
7df7cf6e6d Rename cipherBlock*() functions to cipherBlock*C().
Some of the old names conflict with the new functions that must be created to implement the filter. Rename these to cipherBlock*C() to indicate that they take C-style parameters.

These functions are only used by the Perl LibC code and will be removed or refactored eventually.
2018-11-27 22:49:24 -05:00
David Steele
56ce98b2f0 Explicitly compile with Posix 2001 standard.
This standard was being selectively applied in modules that needed it.

Instead, apply the standard to all compilation for consistency.
2018-11-25 10:06:31 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
1dd06a6e46 Add TlsClient object.
A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.

This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.

Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-21 18:43:25 -05:00
David Steele
bc25db5667 Add interface objects for libxml2.
Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.

This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-20 20:40:11 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
48d2795f31 Merge crypto/random module into crypto/crypto.
There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.
2018-11-06 20:04:16 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
bcca625062 Add bufHex()to Buffer object.
A general-purpose function for converting buffers to hex strings.
2018-09-26 22:33:48 +01:00
David Steele
c49eaec776 Full abstraction of storage driver interface.
The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly.

Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC).

Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
David Steele
fd14ceb399 Rename posix driver files/functions for consistency.
The posix driver was developed over time and the naming is not very consistent.

Rename the files and functions to work well with other drivers and generally favor longer names since the driver functions are seldom (eventually never) used outside the driver itself.
2018-09-13 18:58:22 -04:00
David Steele
77dca5b968 Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added. 2018-08-24 19:31:45 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
e3ff6b209d Filters can now produce output that differs from input.
This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 14:21:53 -04:00
David Steele
01aea0c067 Implement filters that do not modify the buffer.
Update cryptoHash to use the new interface.
2018-07-24 21:08:27 -04:00
David Steele
58e9f1e50c Refactor the common/log tests to not depend on common/harnessLog.
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.

Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
David Steele
0ac176b722 Abstract IO layer out of the storage layer.
This allows the routines to be used for IO objects that do not have a storage representation.

Implement buffer read and write IO objects.
2018-07-19 16:04:20 -04:00
David Steele
cf889790e6 Error if LibC build is performed outside test environment.
LibC is no longer required for production builds.
2018-07-10 15:39:03 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
064ec757e9 Rename cipher module to the more general crypto. 2018-06-11 10:53:16 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
bce3d0fe1c Build libc using links rather than referencing the C files in src directly.
The C library builds with different options which should not be reused for the C binary or vice versa.
2018-05-17 17:09:06 -04:00
David Steele
a9069f3fd3 Begin v2.03 development. 2018-05-09 10:50:05 -04:00
David Steele
ee8aafb3ca v2.02: Parallel Asynchronous Archive Get and Configuration Includes
Bug Fixes:

* Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Fix archive-copy throwing "path not found" error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
* Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
* Fix delta restore failing when a linked file is missing.
* Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)

Features:

* Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
* Add support for additional pgBackRest configuration files in the directory specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang.)
* Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma.)

Improvements:

* Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang.)
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
* Show command parameters as well as command options in initial info log message.
* Rename archive-queue-max option to archive-push-queue-max to avoid confusion with the new archive-get-queue-max option. The old option name will continue to be accepted.
2018-05-06 19:53:42 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00