1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

1768 Commits

Author SHA1 Message Date
David Steele
8c504bd2f9 Improve alignment of expected vs. got error test results.
It is easier to compare the error messages when they start at the same column.
2018-11-09 10:37:59 -05:00
David Steele
b5a3c8c84b Add strCatChr() to String object. 2018-11-09 09:54:55 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
7cf1f42c5c Add time since the beginning of the run to each test statement.
Output the time in seconds of each test statement since the start of the test run.

This helps find individual tests that are running slowly.
2018-11-08 19:50:31 -05:00
David Steele
2150a26424 Storage interface methods no longer declare the driver as const.
This works for the Posix driver but is bad for more dynamic drivers (e.g. S3) that need write access to the driver object.
2018-11-08 18:03:18 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
a9feaba9e5 Add memContextCallbackClear() to prevent double free() calls.
If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.

Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().
2018-11-07 08:51:32 -05:00
David Steele
48d2795f31 Merge crypto/random module into crypto/crypto.
There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.
2018-11-06 20:04:16 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
2cb312ef5a Add cryptoError() and update crypto code to use it.
This adds detail to error messages when available and improves code coverage.
2018-11-06 19:16:00 -05:00
David Steele
bef31f1802 Add base variants to all integer to string conversion functions.
Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.
2018-11-06 18:32:44 -05:00
David Steele
7de9584435 Ignore deleted files in rsync to test/repo.
Deleted files are showing up in git ls-files (added 57d78092) but they don't actually exist on disk.

If there is someway to exclude deleted files from ls-files then I can't find it, so just tell rsync to ignore missing files.
2018-11-04 08:39:48 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
1f8931f732 Improve single test run performance.
Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes).  This simplifies some of the rsync logic and allows make to do what it does best.

Also split build flag files into test, harness, and build to reduce rebuilds.  Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 16:34:04 -04:00
David Steele
7794ab50dc Preserve contents of C unit test build directory between test.pl executions.
The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.

Save the important build flags in a file so the new execution knows whether the build contents can be reused.
2018-11-02 11:56:13 -04:00
David Steele
1a98bd41b4 Mount tmpfs in Vagrantfile instead test.pl.
Mounting/unmounting tmpfs on /home/[user]/test takes time, forces at least 3GB of memory to be available for tests, and makes it harder to preserve data between tests.

Instead, move mounting of tmpfs to the Vagrantfile and add it to fstab so it survives reboots.
2018-11-02 08:37:27 -04:00
Cynthia Shang
34c63276cd Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected.
There are a number of cases where a checksum delta is more appropriate than the default time-based delta:

* Timeline has switched since the prior backup
* File timestamp is older than recorded in the prior backup
* File size changed but timestamp did not
* File timestamp is in the future compared to the start of the backup
* Online option has changed since the prior backup

A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch.  In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention.

Now, a checksum delta will be performed if the backup type is incr/diff.  This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted.

Contributed by Cynthia Shang.
2018-11-01 11:31:25 -04:00
David Steele
cca7a4ffd4 Retry all S3 5xx errors rather than just 500 internal errors.
We were already retrying 500 errors but 503 (rate-limiting) errors were not being retried and would cause an instant failure which aborted the command.

There are only two 5xx errors currently implemented by S3 but instead of adding 503 simply retry all 5xx errors. This is consistent with the http definition of this error class, "the server failed to fulfill an apparently valid request."

Suggested by Craig A. James.
2018-10-30 16:45:42 -04:00
David Steele
286f7e5011 Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
This calculation was missed when the WAL segment size was made dynamic in preparation for PostgreSQL 11.

Fix the calculation by checking the actual WAL file sizes instead of using an estimate based on WAL segment size.  This is more accurate because it takes into account .history and .backup files, which are smaller.  Since the calculation is done in the async process the additional processing time should not adversely affect performance.

Remove the PG_WAL_SIZE constant and instead use local constants where the old value is still required.  This is only the case for some tests and PostgreSQL 8.3 which does not provide a way to get the WAL segment size from pg_control.
2018-10-27 20:00:00 +01:00
David Steele
41b00dc204 Fix issue with archive-push-queue-max not being honored on connection error.
If an error occurred while acquiring a lock on a remote server the error would be reported correctly, but the queue max detection code was not reached.  The tests failed to detect this because they fixed the connection before queue max, allowing the ccde to be reached.

Move the queue max code before the lock so it will run even when remote connections are not working.  This means that no attempt will be made to transfer WAL once queue max has been exceeded, but it makes it much more likely that the code will be reach without error.

Update tests to continue errors up to the point where queue max is exceeded.

Reported by Lardière Sébastien.
2018-10-27 16:57:57 +01:00
David Steele
03b9db9aa2 Fix error after log file open failure when processing should continue.
The C code was warning on failure and continuing but the Perl logging code was never updated with the same feature.

Rather than add the feature to Perl, just disable file logging if the log file cannot be opened.  Log files are always opened by C first, so this will eliminate the error in Perl.

Reported by vthriller.
2018-10-25 14:58:25 +01:00
Cynthia Shang
d301720c58 Change infoArchiveCheckPg() to display the PostgreSQL version as a string in errors.
Previously an integer representation (e.g. 90400) would be displayed which is correct but not as friendly to the user.

Contributed by Cynthia Shang.
2018-10-25 10:48:37 +01:00
Cynthia Shang
5590bfa84c Add tests for InfoPg history fixes introduced in 070455ce.
The existing tests were not adequate to ensure the history was being added in the correct order when some entries were loaded from a file and others added with infoPgAdd().

Contributed by Cynthia Shang.
2018-10-19 18:31:05 +02:00
David Steele
070455ce44 Correct current history item in InfoPg to always be in position 0.
The InfoPg object was partially modified in 960ad732 to place the current history item in position 0, but infoPgDataCurrent() didn't get updated correctly.

Remove this->indexCurrent and make the current position always equal 0.  Use the new lstInsert() function when adding new history items via infoPgAdd(), but continue to use lstAdd() when loading from a file for efficiency.

This does not appear to be a live bug because infoPgDataCurrent() and infoPgAdd() are not yet used in any production code.  The archive-get command is the only C code using InfoPG and it always looks at the entire list of items rather than just the current item.

Suggested by Cynthia Shang.
2018-10-19 12:31:56 +02:00
David Steele
f345db3f7c Add lstInsert() to List object.
Add general purpose insert function and make lstAdd() a special insert case.
2018-10-19 11:52:17 +02:00
Cynthia Shang
09a1ad2c10 Correct cfgDefDataFind() to use UINTP instead of VOIDPP.
Also add the UINTP macro.

Contributed by Cynthia Shang.
2018-10-19 10:51:33 +02:00
David Steele
06d68eada0 Begin v2.07 development. 2018-10-16 17:21:01 +01:00
David Steele
904550c97f v2.06: Checksum Delta Backup and PostgreSQL 11 Support
Bug Fixes:

* Fix missing missing URI encoding in S3 driver. (Reported by Dan Farrell.)
* Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)
* Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.

Features:

* Add checksum delta for incremental backups which uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang.)
* PostgreSQL 11 support, including configurable WAL segment size.

Improvements:

* Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.
* Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Suggested by Douglas J Hunley.)
* Improve support for special characters in filenames.
* Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang.)
2018-10-16 14:56:51 +01:00
David Steele
2c272c220b PostgreSQL 11 support.
PostgreSQL 11 RC1 support was tested in 9ae3d8c46 when the u18 container was rebuilt.  Nothing substantive changed after RC1 so pgBackRest is ready for PostgreSQL 11 GA.
2018-10-15 23:23:49 +01:00
David Steele
9ae3d8c46a Install nodejs from deb.nodesource.com.
The standard npm packages on Ubuntu 18.04 suddenly required libssl1.0 which broke the pgbackrest package builds. Installing nodejs from deb.nodesource.com seems to work fine with standard libssl.

This package is required by ScalityS3 which is used for local S3 testing.
2018-10-15 23:13:08 +01:00
David Steele
98ff8ccc59 Improve documentation in filter.h and filter.internal.h.
When the filter interface internals were split out into a new header file the documentation was not moved as it should have been.  Additionally some functions which should have been moved were left behind.

Move the documentation and functions to filter.internal.h and add more documentation.  Filters are a tricky subject so the more documentation the better.

Also add documentation for the user-facing filter functions in filter.h.
2018-10-09 15:08:49 +01:00
David Steele
68110d04b2 Add ioReadLine()/ioWriteLine() to IoRead/IoWrite objects.
Allow a single linefeed-terminated line to be read or written.  This is useful for various protocol implementations, including HTTP and pgBackRest's protocol.

On read the maximum line size is limited to buffer-size to prevent runaway memory usage in case a linefeed is not found.  This seems fine for HTTP but we may need to revisit this decision when implementing the pgBackRest protocol.  Another option would be to increase the minimum buffer size (currently 16KB).
2018-10-07 17:50:10 +01:00
David Steele
db8dce7adc Disable flapping archive/get unit on CentOS 6.
This test has been flapping since 9b9396c7.  It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs.  It only happens on Travis and is not reproducible in any development environment that we have tried.

For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause.  We are actively developing these tests and hope the issue will be identified during the course of normal development.

A number of improvements were made to the tests while searching for this issue.  While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Stephen Frost
ed5d7a53de Add .gitignore to C src directory.
Suggested by Stephen Frost.
2018-09-30 12:27:05 +01:00
Stephen Frost
3be9f2dee7 Avoid calling strlen() twice in bufNewZ(). 2018-09-30 11:45:37 +01:00
David Steele
5404628148 Fix incorrect error message for duplicate options in configuration files.
Duplicating a non-multi-value option was not throwing the correct message when the option was a boolean.

The reason was that the option was being validated as a boolean before the multi-value check was being done.  The validation code assumed it was operating on a string but was instead operating on a string list causing an assertion to fail.

Since it's not safe to do the multi-value check so late, move it up to the command-line and configuration file parse phases instead.

Reported by Jesper St John.
2018-09-27 17:48:40 +01:00
David Steele
be2271f6d3 Add cfgDefOptionMulti() to identify multi-value options.
Previously this was done in two separate places by checking if an option was type hash or list.

Bad enough that it was in two places, but an upcoming bug fix will add another instance so make it a function.
2018-09-27 17:24:06 +01:00
David Steele
e66e68e324 Add cryptoHmacOne() for HMAC support.
There doesn't seem to be any need to implement this as a filter since current use cases (S3 authentication) work on small datasets.

So, use the single function method provided by OpenSSL for simplicity.
2018-09-27 09:20:47 +01:00
David Steele
6470f03fa2 Comment fixes. 2018-09-26 22:36:42 +01:00
David Steele
bcca625062 Add bufHex()to Buffer object.
A general-purpose function for converting buffers to hex strings.
2018-09-26 22:33:48 +01:00
David Steele
51484a008f Add bufNewZ() to Buffer object.
This constructor creates a Buffer object directly from a zero-terminated string.  The old way was to create a String object first, then convert that to a Buffer using bufNewStr().

Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
David Steele
d038b9a029 Support configurable WAL segment size.
PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB.

There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup.  Both operations require the WAL segment size to properly build a list.

Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database.

The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead.  This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them.  While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL.

Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
David Steele
c0b0b4e541 PostgreSQL 11 Beta 4 support.
Catalog version changed for this release, so update it.

Also update and upload a new container with beta 4 installed.
2018-09-21 13:25:27 -04:00
David Steele
d513a46e6d Update introduction to be pithy. 2018-09-19 18:26:12 -04:00
David Steele
50266cc26c Use command in authorized_hosts to improve SSH security.
Unsecured, passwordless SSH can be a scary thing. If an attacker gains access to one system they can easily hop to other systems.

Add documentation on how to use the command parameter in authorized_keys to limit ssh to running a single command, pgbackrest. There is more that could be done for security but this likely addresses most needs.

Also change references to "trusted ssh" to "passwordless ssh" since this seems more correct.

Suggested by Stephen Frost, Magnus Hagander.
2018-09-19 16:41:20 -04:00
Cynthia Shang
880fbb5e57 Add checksum delta for incremental backups.
Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.

If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.

Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.

Contributed by Cynthia Shang.
2018-09-19 11:12:45 -04:00
Cynthia Shang
bf0691576a Allow delta option to be specified in the pgBackRest configuration file.
This option was previously allowed on the command-line only for no particular reason that we could determine.

Being able to specify it in the config file seems like a good idea and won't change current usage.

Contributed by Cynthia Shang.
2018-09-18 13:29:05 -04:00
Cynthia Shang
b6b2c915b2 Allow hashSize() to run on remote storage.
Apparently we never needed to run this function remotely.

It will be needed by the backup checksum delta feature, so implement it now.

Contributed by Cynthia Shang.
2018-09-18 11:39:48 -04:00
Cynthia Shang
052e483057 Restore bIgnoreMissing flag in backupFile() lost in storage refactor.
The test to make sure that some files (e.g. pg_control) do not get removed during the backup was lost during the storage refactor committed at de7fc37f.

This did not impact the integrity of the backups, but bring it back since it is a nice sanity check.

Contributed by Cynthia Shang.
2018-09-18 10:18:39 -04:00