1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2723 Commits

Author SHA1 Message Date
David Steele
6d11fe53f7 Update RHEL package location (again).
This changed (again) upstream so update the file paths.
2020-04-21 10:51:38 -04:00
David Steele
a999500d2c Remove XML source for test/README.md.
63b9f46a removed test/README.md but did not remove the XML source so it reappeared when the release documentation was built.
2020-04-20 16:21:53 -04:00
David Steele
3e2dc05817 Begin v2.27 development. 2020-04-20 15:58:06 -04:00
David Steele
2352ecf7c0 v2.26: Non-blocking TLS
Bug Fixes:

* Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis.)

Improvements:

* Non-blocking TLS implementation. (Reviewed by Slava Moudry, Cynthia Shang, Stephen Frost.)
* Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)
* TCP keep-alive options are configurable. (Suggested by Marc Cousin.)
* Add io-timeout option.
2020-04-20 15:44:31 -04:00
Cynthia Shang
93e4fe0199 Specify that the io-timeout option is measured in seconds. 2020-04-20 13:11:34 -04:00
David Steele
1c319e051b Remove stray copy of LibCAuto.pm.
This was removed in 79cfd3ae but accidentally added back in 2fa69af8, probably due to an incorrectly resolved merge conflict.
2020-04-20 11:49:11 -04:00
David Steele
4b28f79c92 Clear error queue before each SSL_*() call.
The documentation recommends clearing the error queue before each SSL_*() call.

Since we always check the results of SSL_*() for errors instead of blindly calling SSL_get_error() it's not clear this makes any difference, but it still seems like a good idea to be sure there are no stray errors in the queue.
2020-04-20 11:08:58 -04:00
David Steele
6f41ce1182 Add expire to commands that support io-timeout.
The command list was not complete. Copy the command list from buffer-size to make sure all commands are included.
2020-04-20 09:45:08 -04:00
David Steele
21a1cf9a2f Add sck-block option.
This option puts sockets into blocking mode.

It is only intended for internal use and is not included in the test matrix.
2020-04-17 09:55:40 -04:00
David Steele
5d25e508ae Add io-timeout option.
Timeout used for connections and read/write operations.

Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte.
2020-04-17 09:18:52 -04:00
David Steele
c88684e2bf Non-blocking TLS implementation.
The prior blocking implementation seemed to be prone to locking up on some (especially recent) kernel versions. Since we were unable to reproduce the issue in a development environment we can only speculate as to the cause, but there is a good chance that blocking sockets were the issue or contributed to the issue.

So move to a non-blocking implementation to hopefully clear up these issues. Testing in production environments that were prone to locking shows that the approach is promising and at the very least not a regression.

The main differences from the blocking version are the non-blocking connect() implementation and handling of WANT_READ/WANT_WRITE retries for all SSL*() functions.

Timeouts in the tests needed to be increased because socket connect() and TLS SSL_connect() were not included in the timeout before. The tests don't run any slower, though. In fact, all platforms but Ubuntu 12.04 worked fine with the shorter timeouts.
2020-04-16 16:05:44 -04:00
David Steele
2260a7512a Use poll() instead of select() for monitoring socket read/write ready.
select() is a bit old-fashioned and cumbersome to use. Since the select() code needed to be modified to handle write ready this seems like a good time to upgrade to poll().

poll() has been around for a long time so there doesn't seem to be any need to provide a fallback to select().

Also change the error on timeout from FileReadError to ProtocolError. This works better for read vs. write and failure to poll() is indicative of a protocol error or unexpected EOF.
2020-04-16 15:02:33 -04:00
David Steele
e5e81d3839 Only limit backup copy size for WAL-logged files.
The prior behavior introduced in dcddf3a5 could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup since they are copied via tmp files and could change size during the backup.

In general it seems safer to limit this feature to WAL-logged files which will be reconstructed during recovery.
2020-04-16 14:48:16 -04:00
David Steele
85f4eef55e Add Cirrus CI and Vagrant configuration for FreeBSD and MacOS.
Building on these platforms gives us better coverage for our build code. Cirrus CI was chosen because it is the only service that supports FreeBSD (that we could find).

The FreedBSD configuration for Vagrant is currently just enough to perform a build.

The MacOS configuration is not actually for Vagrant (yet) but does show the steps needed to setup the build environment on MacOS.
2020-04-16 14:13:53 -04:00
David Steele
09fc69f3e8 Make existing Vagrantfile configuration the default.
This allows us to add new configurations mostly without changing the behavior of vagrant from the command line, i.e. vagrant up and vagrant ssh will continue to bring up the default configuration.

However, vagrant destroy -f will remove all configurations. That's really only a change in behavior if more than one configuration is running, which is not currently possible.
2020-04-16 13:58:46 -04:00
David Steele
34d7338604 Fix headers for FreeBSD/MacOS.
It looks like Linux is tolerant of the BSD headers so remove the conditionals.

Eventually it might be a good idea to include these based on configure rules but that seems over overkill for now.
2020-04-16 11:50:35 -04:00
David Steele
492180837e Classify all .h files as C.
Github classifies many C header files as C++, perhaps because they don't contain anything definitively C-like.

Add an override to ensure .h files are always classified as C since this project contains no C++.
2020-04-14 18:03:41 -04:00
David Steele
0c07c5111f Remove empty subexpression from manifest regular expression.
MacOS was not happy about this though other platforms seemed to work fine.

This was likely a typo or a relic from refactoring.
2020-04-14 15:33:43 -04:00
David Steele
8fb0a8fd6e Remove incorrect Github id.
This was a copy-pasto from another contributor entry.
2020-04-14 15:28:43 -04:00
Cynthia Shang
02ebba02b2 Fix typo. 2020-04-14 15:23:19 -04:00
David Steele
9ffa2c618f Use SocketSession/TlsSession for test servers.
A session looks much the same whether it is initiated from the client or the server, so use the session objects to implement the TLS, HTTP, and S3 test servers.

For TLS, at least, there are some differences between client and server sessions so add a client/server type to SocketSession to determine how the session was initiated.

Aside from reducing code duplication, the main advantage is that the test server will now timeout rather than hanging indefinitely when less input that expected is received.
2020-04-14 15:22:49 -04:00
David Steele
71fb28bf3f Always throw error when OpenSSL returns SSL_ERROR_SYSCALL.
Previously an error was only thrown when errno was set but in practice this is usually not the case. This may have something to do with getting errno late but attempts to get it earlier have not been successful. It appears that errno usually gets cleared and spot research seems to indicate that other users have similar issues.

An error at this point indicates unexpected EOF so it seems better to just throw an error all the time and be consistent.

To test this properly our test server needs to call SSL_shutdown() except when the client expects this error.
2020-04-14 15:20:50 -04:00
David Steele
9f2d647bad Split session functionality of TlsClient out into TlsSession.
This abstraction allows the session code to be shared between the TLS client and (upcoming) server code.

Session management is no longer implemented in TlsClient so the HttpClient was updated to free and create sessions as needed. No test changes were required for HttpClient so the functionality should be unchanged.

Mechanical changes to the TLS tests were required to use TlsSession where appropriate rather than TlsClient. There should be no change in functionality other than how sessions are managed, i.e. using tlsClientOpen()/tlsSessionFree() rather than just tlsClientOpen().
2020-04-14 15:02:18 -04:00
David Steele
c9481bb95f Update contributing.xml to use Ubuntu 18.04.
Ubuntu 19.04 was EOL in January and has been removed from the repository servers.

It may be best to stick to LTS versions from now on.
2020-04-14 14:51:14 -04:00
David Steele
f03d1b5b7b Use __noreturn_ on error functions when coverage testing.
The errorInternalThrowSys*() functions were marked as returning during coverage testing even when they had no possibility to return, i.e. the error parameter was set to constant true. This meant the compiler would treat the functions as returning even when they would not.

Instead create completely separate functions for coverage to use for THROW_ON_SYS_ERROR*() that can return and leave the regular functions marked __noreturn__.
2020-04-14 11:43:50 -04:00
David Steele
b7d8d61526 Split session functionality of SocketClient out into SocketSession.
This abstraction allows the session code to be shared between the socket client and (upcoming) server code. There should no difference in how the code works -- only the organization has changed. Note that no changes to the tests were required.

This same abstraction will be required for TlsClient but that will be done in a separate commit because it requires test changes.
2020-04-13 16:59:02 -04:00
Cynthia Shang
b5347070af Remove extraneous assignment.
This is likely leftover cruft from a prior refactor.
2020-04-13 16:44:29 -04:00
David Steele
9d2223b188 Improve comments and fix typos in the storage interface. 2020-04-13 14:57:01 -04:00
Cynthia Shang
310d42ca35 Correct option name in test.pl help. 2020-04-13 14:51:04 -04:00
David Steele
71ce637557 Use standard HARNESS_FORK*() macros to fork test servers.
These forks were done in a custom way (not sure why) and lack the capability of the standard macros for the parent to wait for child exit.

This mean that the server would continue to run after the tests were complete and that multiple servers could run at once. This caused subtle timing and connection issues that required larger timeouts to resolve.

Don't change the timeouts here since they need to be adjusted in future commits anyway.
2020-04-12 09:01:41 -04:00
David Steele
42246401b8 Add comment. 2020-04-11 16:24:45 -04:00
David Steele
674b65886f Suppress uncoverable error in common/io-tls test module.
It is pretty much impossible for a static IP to not resolve to an address but in theory the error could catch other conditions so it seems best to keep it.
2020-04-11 16:21:03 -04:00
David Steele
9a751c2d98 Fix typo. 2020-04-10 12:56:28 -04:00
Cynthia Shang
63b9f46a34 Update contributing documentation and remove test/README.md.
When the Vagrant file was updated to use pgbackrest/ vs /backrest/ as the location for executing tests and building the documentation, parts of the contributing.xml (and hence the CONTRIBUTING.md) were not updated since some parts of the document are not actually executed when the CONTRIBUTING.md is built from contributing.xml: those parts that are executed were updated but those parts that are not executed were not.

This commit fixes the contributing.xml issue but also removes test/README.md as its contents were out of date and redundant given that they are covered in CONTRIBUTING.md.
2020-04-09 18:25:25 -04:00
David Steele
55c3a3c8d3 Allow zero wait times in Wait object.
This limitation forced extra logic in cases where zero wait times were needed.

Remove the limitation and the extra logic in cases where zero wait times are possible.
2020-04-09 18:13:35 -04:00
David Steele
c292e8957d Add some debugging to tests that fork servers.
Help identify whether errors are happening in the forked server or the main test by showing the line number where the server was forked off in the stack trace.
2020-04-07 14:56:09 -04:00
David Steele
099bd85ed7 Reset line number in TEST_ERROR*() macros.
If these are not reset then an error not wrapped in a TEST_ERROR*() macro may show the line number of the previous error in a stack trace, which is confusing.

It is better for the line number to be unreported than wrong.
2020-04-07 14:42:05 -04:00
David Steele
627b495352 Add waitRemaining() to get remaining wait time.
This can be used to set timeouts so they do not wait longer than needed.
2020-04-07 14:35:36 -04:00
David Steele
ac3cfa4c9c Allow default process id to be specified in logInit().
The default process id was previously always 0 but there are cases where it is useful to be able to set the default.

Currently the only use case is for testing but the upcoming server code will also make use of it.
2020-04-07 14:23:31 -04:00
David Steele
2ec379fa40 Add ideator to release note. 2020-04-07 11:10:58 -04:00
David Steele
5e55d58850 Simplify storage driver info and list functions.
The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent.

Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact.

Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs.

Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
David Steele
7679f8f886 Fix issue with THROW*_SYS_ERROR_CODE*() using wrong errno.
When DEBUG_COVERAGE was defined errno was being used instead of the value being passed. This apparently worked by happenstance in the single existing usage but it won't work in general.
2020-04-04 13:59:50 -04:00
David Steele
f984aec665 Standardize some section names in headers.
It's better to start out with plural forms rather than flip back and forth as functions are added and subtracted. So, use "Constructors" instead of "Constructor".

Use "Getters/Setters" rather than "Getters" or "Setters" to avoid similar churn.
2020-04-03 18:15:32 -04:00
David Steele
1aca2cc902 Move extern function comments to headers.
This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them).

Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation.

Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left.

Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc.

A few function prototypes were found that not longer had an implementation. These were removed, but there may be more.

The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
Cynthia Shang
3fbde30c6f Add/remove dependent backups during backup.info reconstruct.
Prior to performing a backup or expiring backups, the backup.info file is validated by reconstructing it from the backups in the repository. When a backup had already been removed from the repo, it was removed from the backup.info file but its dependents were not.

Now, the dependent backups will also be removed from backup.info and only backups in the repo that have their full dependency chain will be added to backup.info if they are missing.
2020-04-03 13:25:38 -04:00
David Steele
f3ae74b0d6 Remove storageRead() and storageWriteDriver().
These functions were only being used in the tests. This usage likely dates to before the include directive was available in define.yaml.
2020-04-03 08:38:28 -04:00
David Steele
1214f1d70b Update RHEL package location.
This changed upstream so update the file paths.
2020-04-03 08:18:07 -04:00
David Steele
713211d89f Clean up const usage in bufPtr() and bufRemainsPtr().
These functions accepted const Buffer objects and returned non-const pointers which is definitely not a good idea. Add bufPtrConst() to handle cases where only a const return value is needed and update call sites.

Use UNCONSTIFY() in cases where library code out of our control requires a non-const pointer. This includes the already-documented exception in command/backup/pageChecksum and input buffers in the gzCompress and gzDecompress filters.
2020-04-02 17:25:49 -04:00
David Steele
76b88a3cd5 Add UNCONSTIFY() macro.
Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers.

Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables).

Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that).

UNCONSTIFY() requires static assert which is a feature in its own right.
2020-04-02 16:58:38 -04:00
David Steele
78beb16d6f Remove unused getters in common/io/write module.
These were probably added for symmetry with IoRead but we would prefer to remove those getters.

So, just remove the equivalents in IoWrite.
2020-04-01 19:33:11 -04:00