1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

965 Commits

Author SHA1 Message Date
David Steele
b6f7cbc315 Add destructors to IoRead and IoWrite objects.
These interfaces previously used the memory context of the object they were associated with and did not have their own destructors.

There are times when it is useful to free the interface without also freeing the underlying object so give IoRead and IoWrite their own memory contexts and destructors.

In passing fix a comment type in bufferRead.c.
2018-11-15 16:25:46 -05:00
David Steele
480e1da798 Add ioWriteFlush() to flush pending output.
By default the IoWrite object does not write until the output buffer is full but this is a problem for protocol messages that must be sent in order to get a response.

ioWriteFlush() is not called internally by IoWrite but can be used at any time to immediately write all bytes from the output buffer without closing the IoWrite object.
2018-11-14 08:53:42 -05:00
David Steele
7918629f59 Modify value of PERL_EMBED_ERROR macro.
Remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:16:53 -05:00
David Steele
504e7d77fa Add comment regarding vars being required in blocks.
Documentation block syntax requires that at least one var be specified.

This limitation should be removed but for now add a comment to describe why a bogus var is defined.
2018-11-14 08:13:01 -05:00
David Steele
14d2f8b86c Reword misleading message in stack trace when parameter buffer is full.
The prior message stated that there had been a buffer overrun which is not true since the code prevents that.

In fact, this message means the parameter buffer filled while building the parameter list. Rather than display a partial list we output this message instead.

Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.
2018-11-14 08:11:11 -05:00
David Steele
a0b2af567a Add TEST_LOG() and TEST_LOG_FMT() macros.
These macros provide a convenient way to output debug information in tests.

They are not intended to be left in test code when it is committed to master.
2018-11-13 20:58:32 -05:00
David Steele
a582ad837f Add logging macros for TimeMSec type. 2018-11-13 20:43:08 -05:00
David Steele
086bc35ddc Make ioReadLine() read less aggressively.
ioReadLine() calls ioRead(), which aggressively tries to fill the output buffer, but this doesn't play well with blocking reads.

Give ioReadLine() an option that tells it to read only what is available.  That doesn't mean the function will never block but at least it won't do so by reading too far.
2018-11-12 21:18:53 -05:00
David Steele
bc810e5a87 Remove error suppression for pgBackRest make.
This looks like a copy-pasto because make works just fine without it.
2018-11-12 20:44:56 -05:00
David Steele
a2672557f8 Allow storage path and file mode to be 0.
Not all storage drivers support Posix-style permissions, e.g. S3.
2018-11-12 20:00:57 -05:00
David Steele
aff9693e64 Add strEmpty() to String object.
strEmpty() is more concise and explanatory than strSize() == 0.
2018-11-12 14:13:55 -05:00
David Steele
38c5f65770 Use THROW_ON_SYS_ERROR macro to improve fork code coverage.
Lack of code coverage in this module is just noise since these functions cannot generally be made to fail.

Any failures are fatal.
2018-11-11 18:12:43 -05:00
David Steele
72ea47bfb3 Add KernelError to report miscellaneous kernel errors. 2018-11-11 18:07:56 -05:00
David Steele
257df96b12 Add THROW*_ON_SYS_ERROR* macros to test and throw system errors.
These macros check the error result internally and are appropriate for system calls that won't return errors and so break coverage.
2018-11-11 18:06:09 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
3cd43fd60c Add new HTML tags and strExtra to DocHtmlElement.
Add HTML tags for table elements.

The strExtra parameter allows adhoc tags to be added to an element for features that can't be implemented with CSS, e.g. colspan.
2018-11-11 16:34:15 -05:00
David Steele
5e3b7cbe6e Apply String constant macros where appropriate.
This may not be all the places where the macros can be useful but it's a good start.
2018-11-10 18:32:49 -05:00
David Steele
8f857a975e Add constant macros to String object.
There are many places (and the number is growing) where a zero-terminated string constant must be transformed into a String object to be usable.  This pattern wastes time and memory, especially since the created string is generally used in a read-only fashion.

Define macros to create constant String objects that are initialized at compile time rather than at run time.
2018-11-10 09:37:12 -05:00
David Steele
df200bee2a Add regExpPrefix() to aid in static prefix searches.
The storageList() command accepts a regular expression as a filter.  This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code.  However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally.

S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible.

Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever.  It requires a ^ anchor and stops scanning when the first special character is found.
2018-11-09 16:50:22 -05:00
David Steele
8c504bd2f9 Improve alignment of expected vs. got error test results.
It is easier to compare the error messages when they start at the same column.
2018-11-09 10:37:59 -05:00
David Steele
b5a3c8c84b Add strCatChr() to String object. 2018-11-09 09:54:55 -05:00
David Steele
3e695af961 New test containers.
* Add libxml2 library needed for S3 development.
* Minor version updates for PostgreSQL.
* Remove PostgreSQL 11 beta/rc repository.
2018-11-08 21:41:41 -05:00
David Steele
7cf1f42c5c Add time since the beginning of the run to each test statement.
Output the time in seconds of each test statement since the start of the test run.

This helps find individual tests that are running slowly.
2018-11-08 19:50:31 -05:00
David Steele
2150a26424 Storage interface methods no longer declare the driver as const.
This works for the Posix driver but is bad for more dynamic drivers (e.g. S3) that need write access to the driver object.
2018-11-08 18:03:18 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
edb2c6eb26 Construct Wait object in milliseconds instead of fractional seconds.
The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.

Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.

Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
David Steele
a9feaba9e5 Add memContextCallbackClear() to prevent double free() calls.
If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.

Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().
2018-11-07 08:51:32 -05:00
David Steele
48d2795f31 Merge crypto/random module into crypto/crypto.
There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.
2018-11-06 20:04:16 -05:00
David Steele
8efa5e6a6a Rename CipherError to CryptoError.
This aligns with the general renaming from cipher to crypto.
2018-11-06 19:38:38 -05:00
David Steele
2cb312ef5a Add cryptoError() and update crypto code to use it.
This adds detail to error messages when available and improves code coverage.
2018-11-06 19:16:00 -05:00
David Steele
bef31f1802 Add base variants to all integer to string conversion functions.
Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.
2018-11-06 18:32:44 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
7794ab50dc Preserve contents of C unit test build directory between test.pl executions.
The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.

Save the important build flags in a file so the new execution knows whether the build contents can be reused.
2018-11-02 11:56:13 -04:00
David Steele
1a98bd41b4 Mount tmpfs in Vagrantfile instead test.pl.
Mounting/unmounting tmpfs on /home/[user]/test takes time, forces at least 3GB of memory to be available for tests, and makes it harder to preserve data between tests.

Instead, move mounting of tmpfs to the Vagrantfile and add it to fstab so it survives reboots.
2018-11-02 08:37:27 -04:00
Cynthia Shang
34c63276cd Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected.
There are a number of cases where a checksum delta is more appropriate than the default time-based delta:

* Timeline has switched since the prior backup
* File timestamp is older than recorded in the prior backup
* File size changed but timestamp did not
* File timestamp is in the future compared to the start of the backup
* Online option has changed since the prior backup

A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch.  In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention.

Now, a checksum delta will be performed if the backup type is incr/diff.  This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted.

Contributed by Cynthia Shang.
2018-11-01 11:31:25 -04:00
David Steele
cca7a4ffd4 Retry all S3 5xx errors rather than just 500 internal errors.
We were already retrying 500 errors but 503 (rate-limiting) errors were not being retried and would cause an instant failure which aborted the command.

There are only two 5xx errors currently implemented by S3 but instead of adding 503 simply retry all 5xx errors. This is consistent with the http definition of this error class, "the server failed to fulfill an apparently valid request."

Suggested by Craig A. James.
2018-10-30 16:45:42 -04:00
David Steele
286f7e5011 Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
This calculation was missed when the WAL segment size was made dynamic in preparation for PostgreSQL 11.

Fix the calculation by checking the actual WAL file sizes instead of using an estimate based on WAL segment size.  This is more accurate because it takes into account .history and .backup files, which are smaller.  Since the calculation is done in the async process the additional processing time should not adversely affect performance.

Remove the PG_WAL_SIZE constant and instead use local constants where the old value is still required.  This is only the case for some tests and PostgreSQL 8.3 which does not provide a way to get the WAL segment size from pg_control.
2018-10-27 20:00:00 +01:00
David Steele
41b00dc204 Fix issue with archive-push-queue-max not being honored on connection error.
If an error occurred while acquiring a lock on a remote server the error would be reported correctly, but the queue max detection code was not reached.  The tests failed to detect this because they fixed the connection before queue max, allowing the ccde to be reached.

Move the queue max code before the lock so it will run even when remote connections are not working.  This means that no attempt will be made to transfer WAL once queue max has been exceeded, but it makes it much more likely that the code will be reach without error.

Update tests to continue errors up to the point where queue max is exceeded.

Reported by Lardière Sébastien.
2018-10-27 16:57:57 +01:00
David Steele
03b9db9aa2 Fix error after log file open failure when processing should continue.
The C code was warning on failure and continuing but the Perl logging code was never updated with the same feature.

Rather than add the feature to Perl, just disable file logging if the log file cannot be opened.  Log files are always opened by C first, so this will eliminate the error in Perl.

Reported by vthriller.
2018-10-25 14:58:25 +01:00
Cynthia Shang
d301720c58 Change infoArchiveCheckPg() to display the PostgreSQL version as a string in errors.
Previously an integer representation (e.g. 90400) would be displayed which is correct but not as friendly to the user.

Contributed by Cynthia Shang.
2018-10-25 10:48:37 +01:00
Cynthia Shang
5590bfa84c Add tests for InfoPg history fixes introduced in 070455ce.
The existing tests were not adequate to ensure the history was being added in the correct order when some entries were loaded from a file and others added with infoPgAdd().

Contributed by Cynthia Shang.
2018-10-19 18:31:05 +02:00
David Steele
070455ce44 Correct current history item in InfoPg to always be in position 0.
The InfoPg object was partially modified in 960ad732 to place the current history item in position 0, but infoPgDataCurrent() didn't get updated correctly.

Remove this->indexCurrent and make the current position always equal 0.  Use the new lstInsert() function when adding new history items via infoPgAdd(), but continue to use lstAdd() when loading from a file for efficiency.

This does not appear to be a live bug because infoPgDataCurrent() and infoPgAdd() are not yet used in any production code.  The archive-get command is the only C code using InfoPG and it always looks at the entire list of items rather than just the current item.

Suggested by Cynthia Shang.
2018-10-19 12:31:56 +02:00
David Steele
f345db3f7c Add lstInsert() to List object.
Add general purpose insert function and make lstAdd() a special insert case.
2018-10-19 11:52:17 +02:00
Cynthia Shang
09a1ad2c10 Correct cfgDefDataFind() to use UINTP instead of VOIDPP.
Also add the UINTP macro.

Contributed by Cynthia Shang.
2018-10-19 10:51:33 +02:00
David Steele
06d68eada0 Begin v2.07 development. 2018-10-16 17:21:01 +01:00
David Steele
904550c97f v2.06: Checksum Delta Backup and PostgreSQL 11 Support
Bug Fixes:

* Fix missing missing URI encoding in S3 driver. (Reported by Dan Farrell.)
* Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)
* Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.

Features:

* Add checksum delta for incremental backups which uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang.)
* PostgreSQL 11 support, including configurable WAL segment size.

Improvements:

* Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.
* Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Suggested by Douglas J Hunley.)
* Improve support for special characters in filenames.
* Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang.)
2018-10-16 14:56:51 +01:00
David Steele
2c272c220b PostgreSQL 11 support.
PostgreSQL 11 RC1 support was tested in 9ae3d8c46 when the u18 container was rebuilt.  Nothing substantive changed after RC1 so pgBackRest is ready for PostgreSQL 11 GA.
2018-10-15 23:23:49 +01:00
David Steele
9ae3d8c46a Install nodejs from deb.nodesource.com.
The standard npm packages on Ubuntu 18.04 suddenly required libssl1.0 which broke the pgbackrest package builds. Installing nodejs from deb.nodesource.com seems to work fine with standard libssl.

This package is required by ScalityS3 which is used for local S3 testing.
2018-10-15 23:13:08 +01:00
David Steele
98ff8ccc59 Improve documentation in filter.h and filter.internal.h.
When the filter interface internals were split out into a new header file the documentation was not moved as it should have been.  Additionally some functions which should have been moved were left behind.

Move the documentation and functions to filter.internal.h and add more documentation.  Filters are a tricky subject so the more documentation the better.

Also add documentation for the user-facing filter functions in filter.h.
2018-10-09 15:08:49 +01:00
David Steele
68110d04b2 Add ioReadLine()/ioWriteLine() to IoRead/IoWrite objects.
Allow a single linefeed-terminated line to be read or written.  This is useful for various protocol implementations, including HTTP and pgBackRest's protocol.

On read the maximum line size is limited to buffer-size to prevent runaway memory usage in case a linefeed is not found.  This seems fine for HTTP but we may need to revisit this decision when implementing the pgBackRest protocol.  Another option would be to increase the minimum buffer size (currently 16KB).
2018-10-07 17:50:10 +01:00