1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

1410 Commits

Author SHA1 Message Date
David Steele
a1a0a23c6a Fix typo. 2020-04-01 16:51:29 -04:00
David Steele
789e364e6b Rename tcp-keep-alive option to sck-keep-alive.
This is really a socket option so the new name is clearer.

Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.
2020-04-01 15:44:51 -04:00
David Steele
5c6fb88bef TCP keep-alive options are configurable.
Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.

Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.

Also move configuration of TCP options before connect() as PostgreSQL does.
2020-03-31 18:13:11 -04:00
David Steele
8989118cc6 Add SocketClient object.
This functionality was embedded into TlsClient but that was starting to get unwieldy.

Add SocketClient to contain all socket-related client functionality.
2020-03-31 12:43:29 -04:00
David Steele
da43db3543 Move common/object.h to common/type/object.h.
This header does not contain a type but is used to define types so this seems like a better location.
2020-03-30 20:52:57 -04:00
David Steele
a29e25a845 Add storage filter performance test.
This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
David Steele
1e0b0c9344 Remove Debian package patch now that it has been merged upstream. 2020-03-29 09:17:39 -04:00
David Steele
bf7b989103 Add time since last test started to test output.
This makes it easier to see the timing of each step in the test.
2020-03-28 18:25:51 -04:00
David Steele
7e519e72d5 Add TEST_TITLE_FMT() macro. 2020-03-28 18:20:29 -04:00
Adrian Vondendriesch
e1c72f6f97 Fix typos. 2020-03-28 17:48:57 -04:00
David Steele
4b1d995bba Update packages required for each CI job.
Documentation builds and tests have only a few packages in common so rearrange packages to save some time and clarify dependencies.

Remove the libperl-dev package which became obsolete when the LibC module was removed in 79cfd3ae.

Add a few comments for good measure.
2020-03-26 22:20:09 -04:00
David Steele
3d255dce3c Add performance/storage test.
The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote.

The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
David Steele
b64755d635 Increase baseline of the performance/type test.
Recent performance improvements allow increasing the baseline of this test.

In general it is best if the baseline is large enough to cause the test to blow up if there are performance regressions.
2020-03-26 20:52:05 -04:00
David Steele
50cf7370ee Add --no-performance to test.pl to suppress performance tests.
Performance tests do not need to be run on all platforms. Using vm=none to run performance tests seems best ... for performance.
2020-03-26 20:36:09 -04:00
David Steele
24e03e1320 Remove RHEL package patch now that it has been merged upstream. 2020-03-26 20:25:42 -04:00
David Steele
fd3dca036b v2.25: LZ4 Compression Support
Features:

* Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari.)

Improvements:

* Improve performance of remote manifest build. (Suggested by Jens Wilke.)
* Fix detection of keepalive options on Linux. (Contributed by Marc Cousin.)
* Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin.)
* Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
* Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
2020-03-26 17:14:53 -04:00
David Steele
8af0462c5d Fix race condition in real/all integration tests.
If the tests are running quickly then the time target might end up the same as the end time of the prior full backup. That means restore auto-select will not pick it as a candidate and restore the last backup instead causing the restore compare to fail.

So, sleep one second.
2020-03-26 15:30:59 -04:00
Cynthia Shang
86f71349ef Improve and centralize backup dependency calculation.
Add functions to select a current backup by label and to retrieve a backup dependency list for any given backup.

Update the expire code to utilize the new functions and to expire backup sets from newest dependency to oldest.
2020-03-26 14:05:40 -04:00
David Steele
e63fdfbfd2 Debug and optimize flag cleanup for unit tests.
Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.

Centralize the logic as much as possible to make it easier to maintain.
2020-03-26 11:16:35 -04:00
David Steele
88d7ee6215 Add srtCatZN().
Append N characters from a zero-terminated string.

Note that the string does not actually need to be zero-terminated as long as N is <= the end of the string being concatenated.
2020-03-25 18:37:35 -04:00
Cynthia Shang
eb7f7dd5ca Fix backup-prior for diff backups in mock/expire integration test.
In the ExpireEnvTest.pm backupCreate() function, backup-prior was incorrectly set for diff backups to the previous backup regardless of what backup type the previous backup was. This did not cause any issues in the Mock Expire tests before because it was not being checked. However, in order to reduce churn in the expect logs for a new feature where the backup-prior is utilized, this is being fixed so that the full backup is always used as backup-prior.
2020-03-25 15:47:03 -04:00
Cynthia Shang
480a11066d More refactoring of command/expire unit test module.
Buffer cleanup following on from e170c53e.
2020-03-25 12:04:23 -04:00
David Steele
d20115d29e Allocate a minimum amount of space when a string is likely to grow.
This helps prevent excessive memory reallocation.
2020-03-25 09:12:51 -04:00
David Steele
ad4187eb9f Improve performance of memResize().
The major bottleneck was finding the memory allocation to be resized since it required a sequential search through a list.

Instead, put the allocation header at the beginning of the allocation and return an offset to the user for their buffer. This allows us to use pointer arithmetic to get back to the allocation header quickly when resizing. A side effect is to make memFree() faster as well. The downside is we won't detect garbage pointers passed to memResize()/memFree(), which is also true for MemContext pointers.

The performance benefits can be pretty large in certain cases, in particular when loading and saving manifests. The following are the before and after performance tests on a 900K file manifest.

Before:

run 003 - manifestNewLoad()/manifestSave()
    000.000s l0125 - generate manifest
    183.411s l0236 -     101.2MB manifest generated with 900000 files
    183.411s l0239 - load manifest
    403.816s l0243 -     completed in 220405ms
    403.816s l0245 -        check file total
    403.816s l0248 - save manifest
    670.217s l0253 -     completed in 266401ms
    670.217s l0256 - find all files
    671.263s l0266 -     completed in 1046ms

After:

run 003 - manifestNewLoad()/manifestSave()
    000.000s l0125 - generate manifest
    007.730s l0236 -     101.2MB manifest generated with 900000 files
    007.730s l0239 - load manifest
    033.431s l0243 -     completed in 25701ms
    033.431s l0245 -        check file total
    033.431s l0248 - save manifest
    057.755s l0253 -     completed in 24324ms
    057.755s l0256 - find all files
    058.689s l0266 -     completed in 934ms
2020-03-24 19:08:00 -04:00
Cynthia Shang
e170c53e7e Refactor command/expire unit test module.
Add titles and use a Buffer to store backup.info instead of a String.
2020-03-23 14:31:04 -04:00
David Steele
f9c86b11a5 More improvements to custom coverage report.
* Fix a few issues with file names being truncated introduced in 787d3fd6.

* Use function line info from the lcov file to calculate which lines to show for uncovered functions.  This is more accurate than what we were doing before and function comment headers are now excluded which reduces clutter in the report.
2020-03-23 12:17:34 -04:00
David Steele
dbb1248bfb Implement TEST_RESULT_*() macros with functions, mostly.
The prior macros had grown over time to be pretty significant pieces of code that required a lot of compile time, though runtime was efficient.

Move most of the macro code into functions to reduce compile time, perhaps at a slight expense to runtime.  The overall performance benefit is 10-15% so this seems like a good tradeoff.

Add TEST_RESULT_UINT_INT() to safely compare uint to int with range checking.
2020-03-22 20:44:51 -04:00
David Steele
d6ffa9ea6d Fix incorrect result types in unit tests.
Upcoming changes to the TEST_RESULT_* macros are more type safe and identified that the wrong macros were being used to test results in many cases.

Commit these changes separately to verify that they work with the current macro versions.

Note that no core bugs were exposed by these changes.
2020-03-22 20:25:31 -04:00
David Steele
e1da6a45e0 Remove TRY...CATCH blocks from TEST_RESULT*() macros.
TRY...CATCH blocks are fairly expensive and when all the TEST_RESULT*() macros succeed they are not needed.

Instead just record info at the start of the result test so a detailed exception can be thrown in test.c in the rare case where an exception occurs.
2020-03-22 16:14:33 -04:00
David Steele
5a8758cc8a Add test function to set stack trace function line number.
This is helpful for test macros that know the line number.

The line number can now be non-zero below the top of the stack without WITH_BACKTRACE so instead ignore the line number for output when it is zero.
2020-03-22 16:04:24 -04:00
David Steele
c2df383aec Fix missing parameter in common/stack-trace test module.
This was passing since we don't test WITH_BACKTRACE in CI because it is used only for test builds.

Ideally we would test this but it doesn't seem worth the trouble at the moment.
2020-03-22 14:24:28 -04:00
David Steele
8c76ea2d93 Fix space lost in d70ca259. 2020-03-22 14:18:16 -04:00
David Steele
06a3f82e91 Add --clean and --clean-only options to test.pl. 2020-03-22 13:46:30 -04:00
David Steele
3ec630f5b2 Allow suppression of times in testing for reproducibility.
Timestamps, timings, etc. cause a lot of log churn when included in documentation.
2020-03-22 10:12:29 -04:00
David Steele
56fb399373 Build contributing documentation on Travis CI.
Building the contributing document has some special requirements because it runs Docker in Docker so the repo path must align on the host and all Docker containers.  Run `pgbackrest/doc/doc.pl` from within the home directory of the user that will do the doc build, e.g. `home/vagrant`.  If the repo is not located directly in the home directory, e.g. `/home/vagrant/pgbackrest`, then a symlink may be used, e.g. `ln -s /path/to/repo /home/vagrant/pgbackrest`.

Mount the repo in the Vagrantfile at /home/vagrant/pgbackrest but provide a link from the old location at /backrest to make the transition less painful.
2020-03-21 18:45:58 -04:00
David Steele
f405c82dcc Don't list directories as changed from the last run.
It's not very useful information and just clutters the list.
2020-03-20 15:00:20 -04:00
David Steele
cfab67a775 Enable coverage testing on Fedora 30.
Now that coverage testing works reliably with gcc9 it makes sense to enable it for CI.
2020-03-20 13:49:23 -04:00
David Steele
782c9f89f4 Remove old coverage data before starting new test.
The old coverage data has been recorded so it is no longer needed. In newer versions of gcc leaving this file around can lead to an error when writing profile data after forking off to a non-pgbackrest binary (which we do in some unit tests).
2020-03-20 13:43:08 -04:00
David Steele
787d3fd67b Improve custom coverage report.
* Show all uncovered branch parts even when there are more than two parts per branch. This is the way gcc9 reports coverage so it needs to work even if it doesn't make as much sense as the old way.

* Show covered branches in functions where coverage is missing.  Showing just the uncovered branches can be confusing because it's not always clear how the coverage relates to the code.  By showing all branch coverage (+ or -) this correspondence is made easier.
2020-03-20 12:54:29 -04:00
David Steele
8af8029006 Fix lcov report when test module missing coverage.
We don't report branch coverage on test modules (e.g. test/src/module/common/errorTest.c) but the code that excluded branch coverage from the test module would also exclude it from all core modules if the test module was included in the lcov report due to lack of function/line coverage.

Adjust the coverage code to only exclude branches during the extraction of test module coverage.
2020-03-20 09:16:38 -04:00
David Steele
f6e9bb0819 Remove obsolete -O2 option for Fedora 30 unit test builds.
For some reason gcc9 would not do -O0 builds in combination with one of the options that libperl required.  Now that libperl is gone this exception is no longer required.
2020-03-19 19:30:09 -04:00
David Steele
2241524c0b Remove obsolete deletes from Vagrantfile.
pgBackRest no longer writes files into the .vagrant path so there's no longer anything to delete.
2020-03-19 18:34:10 -04:00
David Steele
dcddf3a58b Limit backup file copy size to size reported at backup start.
If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data.

This also reduces the likelihood of seeing torn pages during the copy. Torn pages can still occur in the middle of the file, though, so they must be handled.
2020-03-19 13:16:05 -04:00
Cynthia Shang
73315268fd Fix typo. 2020-03-19 12:11:20 -04:00
David Steele
d677b07081 Move coverage code to CoverageTest module.
This code needs some work, which will be easier if it is all in one place.
2020-03-19 12:07:51 -04:00
David Steele
26c89b2c8c Improve testing of files that change size during the backup.
Files can change size during a backup so update and add tests to cover the various scenarios more thoroughly.
2020-03-18 13:40:16 -04:00
David Steele
4ec04e5163 Added redacted manifest to testBackupValidate().
The manifest is excellent for validation but including the entire manifest is too noisy and some values are architecture/algorithm dependent.

Output a redacted version that contains the most important information which can be improved on over time.
2020-03-18 10:10:10 -04:00
David Steele
b8cd1b6790 Add TEST_RESULT_STR_Z_KEYRPL() test macro.
This macro will automatically do key replacement before the comparison. This saves the indentation required for an embedded function call.

Possibly TEST_RESULT_Z_KEYRPL() would also be useful but it will be added when needed.
2020-03-18 10:05:08 -04:00
David Steele
f2548f45ce Allow storage reads to be limited by bytes.
The current use case is reading files from the PostgreSQL cluster during backup.

A file may grow during backup but we only need to copy the number of bytes that were reported during the manifest build.  The rest will be rebuilt from the WAL during recovery so copying more is just a waste of space.

Limiting the copy sizes in backup will be part of a future commit.
2020-03-17 18:16:17 -04:00
David Steele
307e741298 Test that shrunk file is backed up correctly.
It's possible, though rare, for a file to shrink during a backup.

There was no issue with the code but having a test is always a good idea.
2020-03-17 16:01:17 -04:00