1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

937 Commits

Author SHA1 Message Date
David Steele
48d2795f31 Merge crypto/random module into crypto/crypto.
There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.
2018-11-06 20:04:16 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
2cb312ef5a Add cryptoError() and update crypto code to use it.
This adds detail to error messages when available and improves code coverage.
2018-11-06 19:16:00 -05:00
David Steele
bef31f1802 Add base variants to all integer to string conversion functions.
Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.
2018-11-06 18:32:44 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
7794ab50dc Preserve contents of C unit test build directory between test.pl executions.
The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.

Save the important build flags in a file so the new execution knows whether the build contents can be reused.
2018-11-02 11:56:13 -04:00
David Steele
1a98bd41b4 Mount tmpfs in Vagrantfile instead test.pl.
Mounting/unmounting tmpfs on /home/[user]/test takes time, forces at least 3GB of memory to be available for tests, and makes it harder to preserve data between tests.

Instead, move mounting of tmpfs to the Vagrantfile and add it to fstab so it survives reboots.
2018-11-02 08:37:27 -04:00
Cynthia Shang
34c63276cd Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected.
There are a number of cases where a checksum delta is more appropriate than the default time-based delta:

* Timeline has switched since the prior backup
* File timestamp is older than recorded in the prior backup
* File size changed but timestamp did not
* File timestamp is in the future compared to the start of the backup
* Online option has changed since the prior backup

A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch.  In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention.

Now, a checksum delta will be performed if the backup type is incr/diff.  This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted.

Contributed by Cynthia Shang.
2018-11-01 11:31:25 -04:00
David Steele
cca7a4ffd4 Retry all S3 5xx errors rather than just 500 internal errors.
We were already retrying 500 errors but 503 (rate-limiting) errors were not being retried and would cause an instant failure which aborted the command.

There are only two 5xx errors currently implemented by S3 but instead of adding 503 simply retry all 5xx errors. This is consistent with the http definition of this error class, "the server failed to fulfill an apparently valid request."

Suggested by Craig A. James.
2018-10-30 16:45:42 -04:00
David Steele
286f7e5011 Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
This calculation was missed when the WAL segment size was made dynamic in preparation for PostgreSQL 11.

Fix the calculation by checking the actual WAL file sizes instead of using an estimate based on WAL segment size.  This is more accurate because it takes into account .history and .backup files, which are smaller.  Since the calculation is done in the async process the additional processing time should not adversely affect performance.

Remove the PG_WAL_SIZE constant and instead use local constants where the old value is still required.  This is only the case for some tests and PostgreSQL 8.3 which does not provide a way to get the WAL segment size from pg_control.
2018-10-27 20:00:00 +01:00
David Steele
41b00dc204 Fix issue with archive-push-queue-max not being honored on connection error.
If an error occurred while acquiring a lock on a remote server the error would be reported correctly, but the queue max detection code was not reached.  The tests failed to detect this because they fixed the connection before queue max, allowing the ccde to be reached.

Move the queue max code before the lock so it will run even when remote connections are not working.  This means that no attempt will be made to transfer WAL once queue max has been exceeded, but it makes it much more likely that the code will be reach without error.

Update tests to continue errors up to the point where queue max is exceeded.

Reported by Lardière Sébastien.
2018-10-27 16:57:57 +01:00
David Steele
03b9db9aa2 Fix error after log file open failure when processing should continue.
The C code was warning on failure and continuing but the Perl logging code was never updated with the same feature.

Rather than add the feature to Perl, just disable file logging if the log file cannot be opened.  Log files are always opened by C first, so this will eliminate the error in Perl.

Reported by vthriller.
2018-10-25 14:58:25 +01:00
Cynthia Shang
d301720c58 Change infoArchiveCheckPg() to display the PostgreSQL version as a string in errors.
Previously an integer representation (e.g. 90400) would be displayed which is correct but not as friendly to the user.

Contributed by Cynthia Shang.
2018-10-25 10:48:37 +01:00
Cynthia Shang
5590bfa84c Add tests for InfoPg history fixes introduced in 070455ce.
The existing tests were not adequate to ensure the history was being added in the correct order when some entries were loaded from a file and others added with infoPgAdd().

Contributed by Cynthia Shang.
2018-10-19 18:31:05 +02:00
David Steele
070455ce44 Correct current history item in InfoPg to always be in position 0.
The InfoPg object was partially modified in 960ad732 to place the current history item in position 0, but infoPgDataCurrent() didn't get updated correctly.

Remove this->indexCurrent and make the current position always equal 0.  Use the new lstInsert() function when adding new history items via infoPgAdd(), but continue to use lstAdd() when loading from a file for efficiency.

This does not appear to be a live bug because infoPgDataCurrent() and infoPgAdd() are not yet used in any production code.  The archive-get command is the only C code using InfoPG and it always looks at the entire list of items rather than just the current item.

Suggested by Cynthia Shang.
2018-10-19 12:31:56 +02:00
David Steele
f345db3f7c Add lstInsert() to List object.
Add general purpose insert function and make lstAdd() a special insert case.
2018-10-19 11:52:17 +02:00
Cynthia Shang
09a1ad2c10 Correct cfgDefDataFind() to use UINTP instead of VOIDPP.
Also add the UINTP macro.

Contributed by Cynthia Shang.
2018-10-19 10:51:33 +02:00
David Steele
06d68eada0 Begin v2.07 development. 2018-10-16 17:21:01 +01:00
David Steele
904550c97f v2.06: Checksum Delta Backup and PostgreSQL 11 Support
Bug Fixes:

* Fix missing missing URI encoding in S3 driver. (Reported by Dan Farrell.)
* Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)
* Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.

Features:

* Add checksum delta for incremental backups which uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang.)
* PostgreSQL 11 support, including configurable WAL segment size.

Improvements:

* Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.
* Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Suggested by Douglas J Hunley.)
* Improve support for special characters in filenames.
* Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang.)
2018-10-16 14:56:51 +01:00
David Steele
2c272c220b PostgreSQL 11 support.
PostgreSQL 11 RC1 support was tested in 9ae3d8c46 when the u18 container was rebuilt.  Nothing substantive changed after RC1 so pgBackRest is ready for PostgreSQL 11 GA.
2018-10-15 23:23:49 +01:00
David Steele
9ae3d8c46a Install nodejs from deb.nodesource.com.
The standard npm packages on Ubuntu 18.04 suddenly required libssl1.0 which broke the pgbackrest package builds. Installing nodejs from deb.nodesource.com seems to work fine with standard libssl.

This package is required by ScalityS3 which is used for local S3 testing.
2018-10-15 23:13:08 +01:00
David Steele
98ff8ccc59 Improve documentation in filter.h and filter.internal.h.
When the filter interface internals were split out into a new header file the documentation was not moved as it should have been.  Additionally some functions which should have been moved were left behind.

Move the documentation and functions to filter.internal.h and add more documentation.  Filters are a tricky subject so the more documentation the better.

Also add documentation for the user-facing filter functions in filter.h.
2018-10-09 15:08:49 +01:00
David Steele
68110d04b2 Add ioReadLine()/ioWriteLine() to IoRead/IoWrite objects.
Allow a single linefeed-terminated line to be read or written.  This is useful for various protocol implementations, including HTTP and pgBackRest's protocol.

On read the maximum line size is limited to buffer-size to prevent runaway memory usage in case a linefeed is not found.  This seems fine for HTTP but we may need to revisit this decision when implementing the pgBackRest protocol.  Another option would be to increase the minimum buffer size (currently 16KB).
2018-10-07 17:50:10 +01:00
David Steele
db8dce7adc Disable flapping archive/get unit on CentOS 6.
This test has been flapping since 9b9396c7.  It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs.  It only happens on Travis and is not reproducible in any development environment that we have tried.

For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause.  We are actively developing these tests and hope the issue will be identified during the course of normal development.

A number of improvements were made to the tests while searching for this issue.  While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Stephen Frost
ed5d7a53de Add .gitignore to C src directory.
Suggested by Stephen Frost.
2018-09-30 12:27:05 +01:00
Stephen Frost
3be9f2dee7 Avoid calling strlen() twice in bufNewZ(). 2018-09-30 11:45:37 +01:00
David Steele
5404628148 Fix incorrect error message for duplicate options in configuration files.
Duplicating a non-multi-value option was not throwing the correct message when the option was a boolean.

The reason was that the option was being validated as a boolean before the multi-value check was being done.  The validation code assumed it was operating on a string but was instead operating on a string list causing an assertion to fail.

Since it's not safe to do the multi-value check so late, move it up to the command-line and configuration file parse phases instead.

Reported by Jesper St John.
2018-09-27 17:48:40 +01:00
David Steele
be2271f6d3 Add cfgDefOptionMulti() to identify multi-value options.
Previously this was done in two separate places by checking if an option was type hash or list.

Bad enough that it was in two places, but an upcoming bug fix will add another instance so make it a function.
2018-09-27 17:24:06 +01:00
David Steele
e66e68e324 Add cryptoHmacOne() for HMAC support.
There doesn't seem to be any need to implement this as a filter since current use cases (S3 authentication) work on small datasets.

So, use the single function method provided by OpenSSL for simplicity.
2018-09-27 09:20:47 +01:00
David Steele
bcca625062 Add bufHex()to Buffer object.
A general-purpose function for converting buffers to hex strings.
2018-09-26 22:33:48 +01:00
David Steele
51484a008f Add bufNewZ() to Buffer object.
This constructor creates a Buffer object directly from a zero-terminated string.  The old way was to create a String object first, then convert that to a Buffer using bufNewStr().

Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
David Steele
d038b9a029 Support configurable WAL segment size.
PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB.

There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup.  Both operations require the WAL segment size to properly build a list.

Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database.

The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead.  This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them.  While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL.

Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
David Steele
c0b0b4e541 PostgreSQL 11 Beta 4 support.
Catalog version changed for this release, so update it.

Also update and upload a new container with beta 4 installed.
2018-09-21 13:25:27 -04:00
David Steele
d513a46e6d Update introduction to be pithy. 2018-09-19 18:26:12 -04:00
David Steele
50266cc26c Use command in authorized_hosts to improve SSH security.
Unsecured, passwordless SSH can be a scary thing. If an attacker gains access to one system they can easily hop to other systems.

Add documentation on how to use the command parameter in authorized_keys to limit ssh to running a single command, pgbackrest. There is more that could be done for security but this likely addresses most needs.

Also change references to "trusted ssh" to "passwordless ssh" since this seems more correct.

Suggested by Stephen Frost, Magnus Hagander.
2018-09-19 16:41:20 -04:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
bf0691576a Allow delta option to be specified in the pgBackRest configuration file.
This option was previously allowed on the command-line only for no particular reason that we could determine.

Being able to specify it in the config file seems like a good idea and won't change current usage.

Contributed by Cynthia Shang.
2018-09-18 13:29:05 -04:00
Cynthia Shang
b6b2c915b2 Allow hashSize() to run on remote storage.
Apparently we never needed to run this function remotely.

It will be needed by the backup checksum delta feature, so implement it now.

Contributed by Cynthia Shang.
2018-09-18 11:39:48 -04:00
Cynthia Shang
052e483057 Restore bIgnoreMissing flag in backupFile() lost in storage refactor.
The test to make sure that some files (e.g. pg_control) do not get removed during the backup was lost during the storage refactor committed at de7fc37f.

This did not impact the integrity of the backups, but bring it back since it is a nice sanity check.

Contributed by Cynthia Shang.
2018-09-18 10:18:39 -04:00
David Steele
03003562d8 Merge all posix storage tests into a single unit.
As we add storage drivers it's important to keep the tests for each completely separate.  Rather than have three tests for each driver, standardize on having a single test unit for each driver.
2018-09-17 11:45:41 -04:00
David Steele
e55d733041 Add -ftree-coalesce-vars option to unit test compilation.
This is a workaround for inefficient handling of many setjmps in gcc >= 4.9.  Setjmp is used in all error handling, but in the unit tests each test macro contains an error handling block so they add up pretty quickly for large unit tests.

Enabling -ftree-coalesce-vars in affected versions reduces build time and memory requirements by nearly an order of magnitude.  Even so, compiles are much slower than gcc <= 4.8.

We submitted a bug for this at: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87316
Which was marked as a duplicate of: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63155
2018-09-17 11:38:10 -04:00
David Steele
b5f749b21c Add CIFS driver to storage helper for read-only repositories.
For read-only repositories the Posix and CIFS drivers behave exactly the same.  Since that's all we support in C right now it's valid to treat them as the same thing.  An assertion has been added to remind us to add the CIFS driver before allowing the repository to be writable.

Mostly we want to make sure that the C code does not blow up when the repository type is CIFS.
2018-09-16 18:41:30 -04:00
David Steele
a6c346cb04 Clear test directory between test runs.
Previously it was the responsibility of the individual tests to clean up after themselves.  Now the test harness now does the cleanup automatically.

This means that some paths/files need to be recreated with each run but that doesn't happen very often.

An attempt has been made to remove all redundant cleanup code but it's hard to know if everything has been caught.  No issues will be caused by anything that was missed, but they will continue to chew up time in the tests.
2018-09-16 17:26:04 -04:00
David Steele
4119ce208d Move test expect log out of the regular test directory.
Storing the expect log (created by common/harnessLog) in the regular test directory was not ideal.  It showed up in tests and made it difficult to clear the test directory between each run.

Move the expect log to a purpose-built directory one level up so it does not interfere with regular testing.
2018-09-16 15:58:46 -04:00
David Steele
8852622fa2 Fix missing test caused by a misplaced YAML tag. 2018-09-16 15:53:19 -04:00
David Steele
c47fc28c8b Make comment blocks consistent across all tests. 2018-09-16 14:15:21 -04:00
David Steele
84ab787b1a Merge protocol storage helper into storage helper.
These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it.

So, merge the storage helpers. This also has the benefit of making the code a bit simpler.

Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
David Steele
3014b05dab Fix buffer underrun in configuration test harness.
If the total bytes read from the expect log file was 0 then the last byte of whatever was in memory before harnessLogBuffer would be set to 0.

On 32-bit systems this expressed as the high order byte of a pointer being cleared and wackiness (in the form of segfaults) ensued.
2018-09-16 13:41:49 -04:00
David Steele
0c02481d6e Update all interfaces to use variable parameter constructors.
Fixed parameter constructors made adding new interface functions a burden, so we switched to using structs to define interfaces in the storage module at c49eaec7.

While propagating this pattern to the IO interfaces it became obvious that the existing variable parameter function pattern (begun in the storage module) was more succinct and consistent with the existing code.

So, use variable parameter functions to define all interfaces. This assumes that the non-interface parameters will be fixed, which seems reasonable for low-level code.
2018-09-15 21:07:00 -04:00
David Steele
f0ed89f21f Allow C or Perl coverage to run on more than one VM.
C or Perl coverage tests can now be run on any VM provided a recent enough version of Devel::Cover or lcov is available.

For now, leave u18 as the only VM to run coverage tests due to some issues with older versions of lcov.
2018-09-15 13:27:06 -04:00
David Steele
31cdd9d20b Remove compiler warnings that are not valid for u16. 2018-09-15 08:23:55 -04:00
David Steele
c49eaec776 Full abstraction of storage driver interface.
The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly.

Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC).

Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
David Steele
aeb1fa3dfb Don't perform valgrind when requested.
The --no-valgrind flag was not being honored.  It's not clear if this flag ever worked, but it does now.
2018-09-13 19:12:40 -04:00
David Steele
fd14ceb399 Rename posix driver files/functions for consistency.
The posix driver was developed over time and the naming is not very consistent.

Rename the files and functions to work well with other drivers and generally favor longer names since the driver functions are seldom (eventually never) used outside the driver itself.
2018-09-13 18:58:22 -04:00
David Steele
ab1762663c Don't use negations in objects below Storage.
The Storage object represents some some optional parameters as negated if the default is true.  This allows sensible defaults without having to specify most optional parameters.

However, there's no need to propagate this down to functions that require all parameters to be passed -- it makes the code and logging more confusing.  Rename the parameters and update logic to remove negations.
2018-09-13 17:53:48 -04:00
David Steele
5aa458ffae Simplify debug logging by allowing log functions to return String objects.
Previously, debug log functions had to handle NULLs and truncate output to the available buffer size.  This was verbose for both coding and testing.

Instead, create a function/macro combination that allows log functions to return a simple String object.  The wrapper function takes care of the memory context, handles NULLs, and truncates the log string based on the available buffer size.
2018-09-11 18:32:56 -04:00
David Steele
9b9396c7b7 Migrate local, unencrypted, non-S3 archive-get command to C.
The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs.  Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C.

This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible.  It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
David Steele
787e7c295f Change locking around async process forking to be more test friendly.
The lock is now released before the fork and reacquired after the fork so the parent process no longer needs to worry about clearing the lock.

This is the same locking mechanism that will be used once archive-get-async is exec'd as a separate command, so introduce it now to simplify testing.
2018-09-11 14:43:16 -04:00
David Steele
9e574a37dc Make archive-get info messages consistent between C and Perl implementations.
The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all.

Temporarily track the async server status with a flag so that info messages are not output in the async process.  The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 12:30:48 -04:00
Cynthia Shang
cabb3f8577 List allowable values for the buffer-size option in the configuration reference.
Contributed by Cynthia Shang.
Suggested by Stéphane Schildknecht.
2018-09-10 13:33:39 -04:00
Cynthia Shang
e351b8c67c Improve info command to display the stanza cipher type.
Contributed by Cynthia Shang.
Suggested by Douglas J Hunley.
2018-09-10 13:09:45 -04:00
David Steele
c688bc8627 Improve support for special characters in filenames.
% characters caused issues in backup/restore due to filenames being appended directly into a format string.

Reserved XML characters (<>&') caused issues in the S3 driver due to improper escaping.

Add a file with all common special characters to regression testing.
2018-09-10 10:54:34 -04:00
David Steele
80ef6fce75 Fix missing missing URI encoding in S3 driver.
File names with uncommon characters (e.g. @) caused authentication failures due to S3 encoding them correctly while the S3 driver did not.

Reported by Dan Farrell.
2018-09-10 10:47:00 -04:00
David Steele
f7fc8422f7 Make Valgrind return an error even when a non-fatal issue is detected.
By default Valgrind does not exit with an error code when a non-fatal error is detected, e.g. unfreed memory.  Use the --error-exitcode option to enabled this behavior.

Update some minor issues discovered in the tests as a result.  Luckily, no issues were missed in the core code.
2018-09-07 16:50:01 -07:00
David Steele
faaa9a91fd Fix typo in unit test error messages, EXECTED => EXPECTED. 2018-09-07 08:03:56 -07:00
David Steele
6e9b6fdca9 Migrate control functions to detect stop files to C from Perl.
Basic functions to detect the presence of stanza or all stop files and error when they are present.

The functionality to detect stop files without error was not migrated. This functionality is only used by stanza-delete and will be migrated with that command.
2018-09-07 08:03:05 -07:00
David Steele
5bdaa35fa5 Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C.
Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
David Steele
9660076093 Add helper for repository storage.
Implement rules for generating paths within the archive part of the repository. Add a helper function, storageRepo(), to create the repository storage based on configuration settings.

The repository storage helper is located in the protocol module because it will support remote file systems in the future, just as the Perl version does.

Also, improve the existing helper functions a bit using string functions that were not available when they were written.
2018-09-07 07:58:08 -07:00
David Steele
960ad73298 Info objects now parse JSON and use specified storage.
Use JSON code now that it is available and remove temporary hacks used to get things working initially.

Use passed storage objects rather than using storageLocal().  All storage objects in C are still local but this won't always be the case.

Also, move Postgres version conversion functions to postgres/info.c since they have no dependency on the info objects and will likely be useful elsewhere.
2018-09-06 10:12:14 -07:00
David Steele
6361a06181 Fix incorrectly reported error return in info logging.
A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.

Also improve info messages when an archive segment is or is not found.
2018-09-04 21:46:41 -04:00
David Steele
800afeef70 Posix file functions now differentiate between open and missing errors.
The Perl functions do so and the integration tests rely on checking for these errors.  This has been exposed as more functionality is moved into C.

Passing the errors types is now a bit complicated so instead use a flag to determine which errors to throw.
2018-09-04 17:47:23 -04:00
David Steele
375ff9f9d2 Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL.
Previously an error would be generated if other files were present and not owned by the PostgreSQL user.  This hasn't been a big deal in practice but it could cause issues.

Also add tests to make sure the same logic applies with links to files, i.e. all other files in the directory should be ignored.  This was actually working correctly, but there were no tests for it before.
2018-08-31 16:06:40 -04:00
David Steele
41746b53cd Begin v2.06 development. 2018-08-31 14:24:36 -04:00
David Steele
bc7462d86d v2.05: Environment Variable Options and Exclude Temporary/Unlogged Relations
Bug Fixes:

* Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path. This issue did not affect absolute links and relative tablespace links were caught by other checks. (Reported by Cynthia Shang.)
* Remove incompletely implemented online option from the check command. Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly. (Reported by Jason O'Donnell.)
* Fix issue where errors raised in C were not logged when called from Perl. pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging. (Reported by Douglas J Hunley.)
* Fix issue when a boolean option (e.g. delta) was specified more than once. (Reported by Yogesh Sharma.)

Features:

* Allow any option to be set in an environment variable. This includes options that previously could only be specified on the command line, e.g. stanza, and secret options that could not be specified on the command-line, e.g. repo1-s3-key-secret.
* Exclude temporary and unlogged relation (table/index) files from backup. Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced. (Contributed by Cynthia Shang.)
* Allow arbitrary directories and/or files to be excluded from a backup. Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using. (Reviewed by Cynthia Shang.)
* Add log-subprocess option to allow file logging for local and remote subprocesses.
* PostgreSQL 11 Beta 3 support.

Improvements:

* Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. (Contributed by Cynthia Shang.)
* Improve asynchronous archive-get/archive-push performance by directly checking status files. (Contributed by Stephen Frost.)
* Improve error message when a command is missing the stanza option. (Suggested by Sarah Conway.)
2018-08-31 13:19:43 -04:00
David Steele
70514061fd Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path.
Relative link paths were being combined with the paths of previous links (relative or absolute) due to the $strPath variable being modified in the current iteration rather than simply being passed to the next level of recursion.

This issue did not affect absolute links and relative tablespace links were caught by other checks, though the error was confusing.

Reported by Cynthia Shang.
2018-08-30 16:27:36 -04:00
David Steele
c638490451 Documentation updates for exclude feature based on review.
Reviewed by Cynthia Shang.
2018-08-28 16:49:29 -04:00
David Steele
14cde54b37 Limit manifest build recursion (i.e. links followed) to sixteen levels to detect link loops. 2018-08-28 16:27:10 -04:00
David Steele
a6cecf7d5e Prevent manifest from being built more than once. 2018-08-28 16:22:30 -04:00
Stephen Frost
4498f726a5 Improve asynchronous archive-get/archive-push performance by directly checking status files.
Prior to this commit, an expression was used to search the spool directory for ok/error files for a specific WAL segment. This involved setting up a regular expression and using opendir/readdir.

Instead, directly probe for the status files, checking directly if a '.ok' or '.error' file exists, avoiding the regular expression and eliminating the directory scan.

Only the two files now probed for could have ever matched the regular expression which had been provided and it's unlikely that many more additional files will be added, so this is a good improvement, and optimization, with little downside.

Contributed by Stephen Frost.
2018-08-28 16:04:53 -04:00
David Steele
bef58a7974 Allow arbitrary directories and/or files to be excluded from a backup.
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
2018-08-27 15:51:05 -04:00
David Steele
77dca5b968 Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added. 2018-08-24 19:31:45 -04:00
David Steele
4c3cf435c0 Add cvtZToUInt() to convert string to unsigned int. 2018-08-24 17:08:00 -04:00
Cynthia Shang
eb30d88b6a Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta.
Contributed by Cynthia Shang.
2018-08-24 16:50:33 -04:00
David Steele
0ed37ab9e7 Update Archive::Info->archiveIdList() to return a valid error code instead of unknown. 2018-08-24 12:13:10 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
512da125ad Improve error reporting for TEST_ASSIGN() and TEST_RESULT_VOID() macros. 2018-08-22 19:03:13 -04:00
David Steele
ad989068d2 Fix issue when a boolean option (e.g. delta) was specified more than once.
Reported by Yogesh Sharma.
2018-08-22 16:45:25 -04:00
David Steele
8a8738308c Enable -Wvla. 2018-08-22 14:48:37 -04:00
David Steele
de5614db6b PostgreSQL 11 Beta 3 support. 2018-08-15 14:05:39 -04:00
David Steele
d0b9f986a0 Allow any option to be set in an environment variable.
This includes options that previously could only be specified on the command line, e.g. stanza.
2018-08-15 10:52:53 -04:00
David Steele
4a822d3032 Correct OptionInvalidError to OptionInvalidValueError in boolean option parsing. 2018-08-15 10:32:18 -04:00
David Steele
cb4b715533 Add strReplaceChr() to String object. 2018-08-14 16:49:38 -04:00
David Steele
4a176681c3 Add cvtCharToZ() and macro for debugging char params. 2018-08-14 16:18:17 -04:00
David Steele
9e3273fdf9 Define cipher magic size with sizeof() rather than using a constant. 2018-08-14 16:08:58 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
e3ff6b209d Filters can now produce output that differs from input.
This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 14:21:53 -04:00
David Steele
40924e8f5c Fix invalid log level in log-path option reference.
Reported by Camilo Aguilar.
2018-08-13 07:47:26 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
cb6821e853 Stop trying to arrange contributors in release.xml by last/first name.
Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense.

Inspired by Christophe Pettus' PostgresOpen 2017 talk, "Human Beings Do Not Have a Primary Key".
2018-08-12 18:19:34 -04:00
David Steele
cd5df3570b Validate configuration options in a single pass.
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
2018-08-11 12:55:33 -04:00
David Steele
f06bf9e832 Improve error message when a command is missing the stanza option.
Suggested by Sarah Conway.
2018-08-11 09:47:07 -04:00
Cynthia Shang
8ab2e72960 Migrate minimum set of code for reading archive.info files from Perl to C.
Contributed by Cynthia Shang.
2018-08-09 08:57:21 -04:00
David Steele
7993f1a966 Add basic C JSON parser. 2018-08-09 08:06:23 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
ac16cb7ff1 Add cvtBoolToConstZ() to simplify conversion of boolean to string. 2018-07-28 09:39:52 -04:00
David Steele
7a00bbebfa Do nothing in memContextMove() when the context is already in the specified parent. 2018-07-28 09:04:06 -04:00
David Steele
cb975a5330 Remove redundant lines from embedded Perl by combining blank lines. 2018-07-28 08:43:58 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
01aea0c067 Implement filters that do not modify the buffer.
Update cryptoHash to use the new interface.
2018-07-24 21:08:27 -04:00
David Steele
8e1017c162 Move most host setup to containers defined in the documentation.
This includes PostgreSQL installation which had previously been included in the documentation. This way produces faster builds and there is no need for us to document PostgreSQL installation.
2018-07-23 07:59:24 -04:00
David Steele
2723831032 Allow containers to be defined in a document.
The defined containers are built before the document build begins which allows them to be reused.
2018-07-23 07:49:15 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
58e9f1e50c Refactor the common/log tests to not depend on common/harnessLog.
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.

Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
David Steele
1359e2908c Fix issue where errors raised in C were not logged when called from Perl.
pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging.

Reported by Douglas J Hunley.
2018-07-20 08:11:34 -04:00
David Steele
d3cfeebdf9 Rename error-handling variables in Main.pm to conform to standard. 2018-07-20 08:03:44 -04:00
David Steele
fa53e2eddb Show exact log level required for stack trace param output instead of just "debug". 2018-07-20 07:15:28 -04:00
David Steele
0ac176b722 Abstract IO layer out of the storage layer.
This allows the routines to be used for IO objects that do not have a storage representation.

Implement buffer read and write IO objects.
2018-07-19 16:04:20 -04:00
David Steele
5dc8a2ec08 storageFileRead() accepts a buffer for output rather than creating one.
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
2018-07-17 19:01:54 -04:00
Cynthia Shang
0acf705416 Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
Contributed by Cynthia Shang.
2018-07-16 17:25:15 -04:00
Cynthia Shang
4e38cbaea9 Add iniSectionList() to Ini object and remove dead code.
Contributed by Cynthia Shang.
2018-07-12 15:28:46 -04:00
Cynthia Shang
0e6b927a17 Add uint64 variant type and supporting conversion functions.
Contributed by Cynthia Shang.
Reviewed by Stephen Frost.
2018-07-12 15:23:18 -04:00
David Steele
0e331b12ba Improve performance of string to int conversion.
Use strtoll() instead of sprintf() for conversion. Also use available integer min/max constants rather than hard-coded values.

Reviewed by Stephen Frost.
Suggested by Stephen Frost.
2018-07-12 15:15:32 -04:00
David Steele
cf889790e6 Error if LibC build is performed outside test environment.
LibC is no longer required for production builds.
2018-07-10 15:39:03 -04:00
David Steele
b1bc53657d Begin v2.05 development. 2018-07-09 08:15:16 -04:00
David Steele
a8143ec125 v2.04: Critical Bug Fix for Backup Resume
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.

Bug Fixes:

* Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
* Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reported by Nj Baliyan.)
* Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz.)

Features:

* PostgreSQL 11 Beta 2 support.

Improvements:

* Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
* Set search_path = 'pg_catalog' on PostgreSQL connections. (Suggested by Stephen Frost.)
2018-07-05 20:16:41 -04:00
David Steele
6f6dad6dd5 Remove call to lscpu which can vary widely by build host. 2018-07-05 16:49:46 -04:00
David Steele
22adb395db Add zero-length file to mock/all test. 2018-07-05 15:40:50 -04:00
David Steele
db17973cd0 Fix critical bug in resume that resulted in inconsistent backups.
A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".

Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 14:01:57 -04:00
David Steele
3d07e01fe9 Make ls ordering deterministic in mock/all test. 2018-07-01 20:02:37 -04:00
Andrew Schwartz
1bd98b61df Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.
AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.

Fixed by Andrew Schwartz.
2018-07-01 08:17:27 -04:00
David Steele
7e65ddad34 PostgreSQL 11 Beta 2 support. 2018-06-30 14:55:25 -04:00
David Steele
7b0e65d488 Improve the HTTP client to set content-length to 0 when not specified by the server.
S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.

Suggested by Adam K. Sumner.
2018-06-26 17:27:22 -04:00
David Steele
e8c68b9c66 Update parameters for VBoxService start. 2018-06-26 16:56:05 -04:00
David Steele
ead70e16bb Add sample S3 policy to restrict bucket privileges.
Suggested by Douglas J Hunley, Jason O'Donnell.
2018-06-23 13:09:56 -04:00
David Steele
aa41e00c9c Set search_path = 'pg_catalog' on PostgreSQL connections.
Suggested by Stephen Frost.
2018-06-21 11:39:37 -04:00
David Steele
7fee739710 Remove Debian package patch since it has been committed upstream. 2018-06-20 18:26:53 -04:00
David Steele
6084e9be04 Fix default location of pgbackrest.conf in option reference. 2018-06-16 13:07:01 -04:00
David Steele
1a0d568600 Preliminary documentation for PostgreSQL 11 unprivileged user backup. 2018-06-14 19:05:35 -04:00
David Steele
3793ae1e4f Convert the not very portable uint type to unsigned int.
Suggested by Devrim Gündüz.
2018-06-14 17:59:10 -04:00
Cynthia Shang
4e7692b74d Fix error in selective restore when only one user database exists in the cluster.
Fixed by Cynthia Shang.
Reported by Nj Baliyan.
2018-06-14 17:44:44 -04:00
David Steele
fbd1da0523 Create a new section to describe building pgBackRest and build on a separate host. 2018-06-13 13:43:17 -04:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
064ec757e9 Rename cipher module to the more general crypto. 2018-06-11 10:53:16 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
5c0099781d Update Debian package to add debug symbols to pgBackRest executable. 2018-06-06 22:19:01 -04:00