1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2673 Commits

Author SHA1 Message Date
David Steele
f19c59ab38 Remove unused exports from pgBackRestBuild::Config::Data.
These were once used heavily in the Perl code but now only a few of them are used in the documentation.

Remove the unused exports as there is generally no need create them for new commands/options.
2020-04-01 18:03:16 -04:00
David Steele
3aedcd1776 Enable FD_CLOEXEC.
PostgreSQL enables this option when available which seems like a good idea since we also do not share connections between processes.

Note that as in PostgreSQL there is no way to disable this option.
2020-04-01 17:20:47 -04:00
David Steele
967f2c0d7f Enable TCP_NODELAY.
PostgreSQL enables this option when available which seems like a good idea since we also buffer transmissions.

Note that as in PostgreSQL there is no way to disable this option.
2020-04-01 16:56:15 -04:00
David Steele
a1a0a23c6a Fix typo. 2020-04-01 16:51:29 -04:00
David Steele
789e364e6b Rename tcp-keep-alive option to sck-keep-alive.
This is really a socket option so the new name is clearer.

Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.
2020-04-01 15:44:51 -04:00
David Steele
5c6fb88bef TCP keep-alive options are configurable.
Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.

Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.

Also move configuration of TCP options before connect() as PostgreSQL does.
2020-03-31 18:13:11 -04:00
David Steele
8989118cc6 Add SocketClient object.
This functionality was embedded into TlsClient but that was starting to get unwieldy.

Add SocketClient to contain all socket-related client functionality.
2020-03-31 12:43:29 -04:00
David Steele
abb7639807 Add OBJECT_DEFINE_GET() macro to define object getters.
Simple object getters involve a lot of boilerplate code so create a macro to simplify implementing them.
2020-03-31 11:10:26 -04:00
David Steele
da43db3543 Move common/object.h to common/type/object.h.
This header does not contain a type but is used to define types so this seems like a better location.
2020-03-30 20:52:57 -04:00
David Steele
a29e25a845 Add storage filter performance test.
This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
David Steele
1e0b0c9344 Remove Debian package patch now that it has been merged upstream. 2020-03-29 09:17:39 -04:00
David Steele
bf7b989103 Add time since last test started to test output.
This makes it easier to see the timing of each step in the test.
2020-03-28 18:25:51 -04:00
David Steele
7e519e72d5 Add TEST_TITLE_FMT() macro. 2020-03-28 18:20:29 -04:00
Adrian Vondendriesch
e1c72f6f97 Fix typos. 2020-03-28 17:48:57 -04:00
Marc Cousin
af734d69f4 Restore prefix/exec_prefix Makefile variables removed in 237a3da4. 2020-03-27 08:22:58 -04:00
David Steele
4b1d995bba Update packages required for each CI job.
Documentation builds and tests have only a few packages in common so rearrange packages to save some time and clarify dependencies.

Remove the libperl-dev package which became obsolete when the LibC module was removed in 79cfd3ae.

Add a few comments for good measure.
2020-03-26 22:20:09 -04:00
David Steele
65285ff5b2 Rearrange Travis CI jobs based on current runtimes.
The runtimes of the tests have change with recent updates and it is generally best to order the slowest tests first.
2020-03-26 22:00:40 -04:00
David Steele
0f742204e8 Add comments to make the purpose of each CI job clearer. 2020-03-26 21:39:40 -04:00
David Steele
09d26d54ce Updates for key changes in travis.yaml.
The format has evolved over time and the old keys were showing warnings during builds.
2020-03-26 21:16:21 -04:00
David Steele
3d255dce3c Add performance/storage test.
The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote.

The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
David Steele
b64755d635 Increase baseline of the performance/type test.
Recent performance improvements allow increasing the baseline of this test.

In general it is best if the baseline is large enough to cause the test to blow up if there are performance regressions.
2020-03-26 20:52:05 -04:00
David Steele
50cf7370ee Add --no-performance to test.pl to suppress performance tests.
Performance tests do not need to be run on all platforms. Using vm=none to run performance tests seems best ... for performance.
2020-03-26 20:36:09 -04:00
David Steele
24e03e1320 Remove RHEL package patch now that it has been merged upstream. 2020-03-26 20:25:42 -04:00
David Steele
fa86422371 Begin v2.26 development. 2020-03-26 17:20:58 -04:00
David Steele
fd3dca036b v2.25: LZ4 Compression Support
Features:

* Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari.)

Improvements:

* Improve performance of remote manifest build. (Suggested by Jens Wilke.)
* Fix detection of keepalive options on Linux. (Contributed by Marc Cousin.)
* Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin.)
* Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
* Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
2020-03-26 17:14:53 -04:00
David Steele
8af0462c5d Fix race condition in real/all integration tests.
If the tests are running quickly then the time target might end up the same as the end time of the prior full backup. That means restore auto-select will not pick it as a candidate and restore the last backup instead causing the restore compare to fail.

So, sleep one second.
2020-03-26 15:30:59 -04:00
Cynthia Shang
86f71349ef Improve and centralize backup dependency calculation.
Add functions to select a current backup by label and to retrieve a backup dependency list for any given backup.

Update the expire code to utilize the new functions and to expire backup sets from newest dependency to oldest.
2020-03-26 14:05:40 -04:00
Cynthia Shang
048a36ea39 Order commands alphabetically in definition of set option. 2020-03-26 13:04:59 -04:00
David Steele
2c2bc8a32b Debug and optimize flag cleanup for unit tests.
Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.

Centralize the logic as much as possible to make it easier to maintain.
2020-03-26 12:16:22 -04:00
David Steele
e63fdfbfd2 Debug and optimize flag cleanup for unit tests.
Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.

Centralize the logic as much as possible to make it easier to maintain.
2020-03-26 11:16:35 -04:00
David Steele
d242114dc0 Rough script to add tables to the user guide for performance tests.
This really needs to be worked on but for now at least preserve to code to be improved later.
2020-03-25 19:00:09 -04:00
David Steele
10b4b9af87 Improve performance of jsonToStrInternal() and jsonFromStrInternal().
Most strings do not contain escape/d characters, so optimize the path where the string contains few or no escape/d characters.

This means far fewer calls to strCatChr() in favor of strCatZN(), which is much more efficient.
2020-03-25 18:52:20 -04:00
David Steele
88d7ee6215 Add srtCatZN().
Append N characters from a zero-terminated string.

Note that the string does not actually need to be zero-terminated as long as N is <= the end of the string being concatenated.
2020-03-25 18:37:35 -04:00
David Steele
b10270eee8 Use conditional in strPtr() to encourage inlining.
This construct is logically equivalent but it seems to make the compiler more likely to inline the function, probably because the compiled code is slightly smaller.
2020-03-25 16:03:33 -04:00
Cynthia Shang
eb7f7dd5ca Fix backup-prior for diff backups in mock/expire integration test.
In the ExpireEnvTest.pm backupCreate() function, backup-prior was incorrectly set for diff backups to the previous backup regardless of what backup type the previous backup was. This did not cause any issues in the Mock Expire tests before because it was not being checked. However, in order to reduce churn in the expect logs for a new feature where the backup-prior is utilized, this is being fixed so that the full backup is always used as backup-prior.
2020-03-25 15:47:03 -04:00
David Steele
4c31a922a3 Remove backup after timeline switch from user guide.
It is no longer necessary (since 2.07, 34c63276) to do a backup manually after a timeline switch.
2020-03-25 12:17:20 -04:00
Cynthia Shang
480a11066d More refactoring of command/expire unit test module.
Buffer cleanup following on from e170c53e.
2020-03-25 12:04:23 -04:00
David Steele
d20115d29e Allocate a minimum amount of space when a string is likely to grow.
This helps prevent excessive memory reallocation.
2020-03-25 09:12:51 -04:00
David Steele
ad4187eb9f Improve performance of memResize().
The major bottleneck was finding the memory allocation to be resized since it required a sequential search through a list.

Instead, put the allocation header at the beginning of the allocation and return an offset to the user for their buffer. This allows us to use pointer arithmetic to get back to the allocation header quickly when resizing. A side effect is to make memFree() faster as well. The downside is we won't detect garbage pointers passed to memResize()/memFree(), which is also true for MemContext pointers.

The performance benefits can be pretty large in certain cases, in particular when loading and saving manifests. The following are the before and after performance tests on a 900K file manifest.

Before:

run 003 - manifestNewLoad()/manifestSave()
    000.000s l0125 - generate manifest
    183.411s l0236 -     101.2MB manifest generated with 900000 files
    183.411s l0239 - load manifest
    403.816s l0243 -     completed in 220405ms
    403.816s l0245 -        check file total
    403.816s l0248 - save manifest
    670.217s l0253 -     completed in 266401ms
    670.217s l0256 - find all files
    671.263s l0266 -     completed in 1046ms

After:

run 003 - manifestNewLoad()/manifestSave()
    000.000s l0125 - generate manifest
    007.730s l0236 -     101.2MB manifest generated with 900000 files
    007.730s l0239 - load manifest
    033.431s l0243 -     completed in 25701ms
    033.431s l0245 -        check file total
    033.431s l0248 - save manifest
    057.755s l0253 -     completed in 24324ms
    057.755s l0256 - find all files
    058.689s l0266 -     completed in 934ms
2020-03-24 19:08:00 -04:00
David Steele
98e5affa8a Free temp mem context periodically in storageRemoteInfoList().
Other storage*InfoList() functions do this but it was missed here.

memResize()/memFree() operations become more expensive as the mem context grows larger so freeing it periodically saves processing time.
2020-03-24 18:44:13 -04:00
Cynthia Shang
e170c53e7e Refactor command/expire unit test module.
Add titles and use a Buffer to store backup.info instead of a String.
2020-03-23 14:31:04 -04:00
David Steele
f9c86b11a5 More improvements to custom coverage report.
* Fix a few issues with file names being truncated introduced in 787d3fd6.

* Use function line info from the lcov file to calculate which lines to show for uncovered functions.  This is more accurate than what we were doing before and function comment headers are now excluded which reduces clutter in the report.
2020-03-23 12:17:34 -04:00
David Steele
dbb1248bfb Implement TEST_RESULT_*() macros with functions, mostly.
The prior macros had grown over time to be pretty significant pieces of code that required a lot of compile time, though runtime was efficient.

Move most of the macro code into functions to reduce compile time, perhaps at a slight expense to runtime.  The overall performance benefit is 10-15% so this seems like a good tradeoff.

Add TEST_RESULT_UINT_INT() to safely compare uint to int with range checking.
2020-03-22 20:44:51 -04:00
David Steele
d6ffa9ea6d Fix incorrect result types in unit tests.
Upcoming changes to the TEST_RESULT_* macros are more type safe and identified that the wrong macros were being used to test results in many cases.

Commit these changes separately to verify that they work with the current macro versions.

Note that no core bugs were exposed by these changes.
2020-03-22 20:25:31 -04:00
David Steele
e1da6a45e0 Remove TRY...CATCH blocks from TEST_RESULT*() macros.
TRY...CATCH blocks are fairly expensive and when all the TEST_RESULT*() macros succeed they are not needed.

Instead just record info at the start of the result test so a detailed exception can be thrown in test.c in the rare case where an exception occurs.
2020-03-22 16:14:33 -04:00
David Steele
5a8758cc8a Add test function to set stack trace function line number.
This is helpful for test macros that know the line number.

The line number can now be non-zero below the top of the stack without WITH_BACKTRACE so instead ignore the line number for output when it is zero.
2020-03-22 16:04:24 -04:00
David Steele
bd461a1899 Add TestError for testing.
This error type distinguishes test errors with detailed debug info from normal errors which may need, for example, a stack trace to be added for context.
2020-03-22 14:40:36 -04:00
David Steele
c2df383aec Fix missing parameter in common/stack-trace test module.
This was passing since we don't test WITH_BACKTRACE in CI because it is used only for test builds.

Ideally we would test this but it doesn't seem worth the trouble at the moment.
2020-03-22 14:24:28 -04:00
David Steele
8c76ea2d93 Fix space lost in d70ca259. 2020-03-22 14:18:16 -04:00
David Steele
4c831d8e83 Use --clean-only for reproducible builds in contributing documentation.
If the work or result directories already contain data then the docs might be generated slightly differently.  Doing a clean ensures they will always produce the same output (provided the code does not change).
2020-03-22 13:50:31 -04:00