1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

1646 Commits

Author SHA1 Message Date
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
David Steele
64a21920e2 Move S3 initialization in user guide quickstart.
The previous location was too late to allow --var=s3-all=y to work with --require=/repo-host, which depends on /quickstart/configure-archiving.

Since the section is not included in production documentation, the position is not very important to flow so just move it to where it works.
2020-05-03 18:42:33 -04:00
David Steele
ef93249922 Add contributor for 816ba924 and reclassify as a bug. 2020-05-01 17:32:31 -04:00
David Steele
816ba9244f Allow pg-path1 to be optional for synchronous archive-push.
If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control.

Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.
2020-05-01 10:30:35 -04:00
David Steele
baf8cb9068 Fix issue checking if file links are contained in path links.
There is no conflict if the path containing a file link is a parent path of a path link. The Perl code apparently had this right but the migration to C missed it.

Exclude this case when checking for link conflicts.
2020-04-30 10:47:09 -04:00
David Steele
ce55866714 Enforce non-null for most string options.
There have been a number of segfaults reported because a string option expected to be non-null was actually null. This is generally due to options that are expected to be set but are in fact optional.

Protect against this by creating cfgOptionStrNull() to get options that can be null, while changing cfgOptionStr() to always expect non-null. There are relatively few places where nulls are expected.

There is definitely a chance for breakage here as null options might currently be working in the field but will be caught by this new check. Hopefully introducing the check early in the release cycle will allow us to catch any issues.
2020-04-30 10:34:44 -04:00
Cynthia Shang
c5241e5007 Expire WAL archive only when repo-retention-archive threshold is met.
Previously when retention-archive was set (either by the user or by default), archives prior to the archive-start of the oldest remaining full backup (after backup expiration occurred) would be expired even though the retention-archive threshold had not been met. For example, if there were 1 full backup remaining after backup expiration and the retention-archive was set to 2 and retention-archive-type=full, then archives prior to the archive-start of the remaining full backup would still be removed even though retention-archive required 2 full backups remaining before archives should be expired.

The thought was to keep the archive directory clean and since the full backup did not require prior archives, it was safe to delete them. However, this has caused problems for some users in the past (because they needed the WAL for other purposes) and with the new adhoc and time-based retention features, it was decided that the archives should remain until the threshold was met. The archives will eventually be removed and if having them causes space issues, the expire command and the retention-archive can always be run and adjusted.
2020-04-29 08:06:49 -04:00
Cynthia Shang
1c1a710460 Add --set option to the expire command.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 14:00:36 -04:00
Cynthia Shang
483838233f The expire command now checks if a stop file is present.
This check has always been missing but it was not noticed because expire is not usually run independently of backup, which does the check correctly.
2020-04-27 10:57:19 -04:00
Cynthia Shang
a5b2630d7c Throw user-friendly error if expire is not run on repository host.
Running the expire command on a non-repository host threw an assertion because the command is not yet configured for remote execution.
2020-04-27 10:49:05 -04:00
David Steele
12069ff8f3 Add reviewer. 2020-04-24 08:45:54 -04:00
David Steele
0dd6629a2d Automatically add default user as reviewer.
Contributions looked like they had no reviewer when reviewed by the default user.
2020-04-24 08:40:54 -04:00
Stefan Fercot
e92eb709d6 Add backup/expire running status to the info command.
This is implemented by checking for a backup lock on the host where info is running so there are a few limitations:

* It is not currently possible to know which command is running: backup, expire, or stanza-*. The stanza commands are very unlikely to be running so it's pretty safe to guess backup/expire. Command information may be added to the lock file to improve the accuracy of the reported command.

* If the info command is run on a host that is not participating in the backup, e.g. a standby, then there will be no backup lock. This seems like a minor limitation since running info on the repo or primary host is preferred.
2020-04-24 08:00:00 -04:00
David Steele
a999500d2c Remove XML source for test/README.md.
63b9f46a removed test/README.md but did not remove the XML source so it reappeared when the release documentation was built.
2020-04-20 16:21:53 -04:00
David Steele
3e2dc05817 Begin v2.27 development. 2020-04-20 15:58:06 -04:00
David Steele
2352ecf7c0 v2.26: Non-blocking TLS
Bug Fixes:

* Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis.)

Improvements:

* Non-blocking TLS implementation. (Reviewed by Slava Moudry, Cynthia Shang, Stephen Frost.)
* Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)
* TCP keep-alive options are configurable. (Suggested by Marc Cousin.)
* Add io-timeout option.
2020-04-20 15:44:31 -04:00
Cynthia Shang
93e4fe0199 Specify that the io-timeout option is measured in seconds. 2020-04-20 13:11:34 -04:00
David Steele
5d25e508ae Add io-timeout option.
Timeout used for connections and read/write operations.

Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte.
2020-04-17 09:18:52 -04:00
David Steele
c88684e2bf Non-blocking TLS implementation.
The prior blocking implementation seemed to be prone to locking up on some (especially recent) kernel versions. Since we were unable to reproduce the issue in a development environment we can only speculate as to the cause, but there is a good chance that blocking sockets were the issue or contributed to the issue.

So move to a non-blocking implementation to hopefully clear up these issues. Testing in production environments that were prone to locking shows that the approach is promising and at the very least not a regression.

The main differences from the blocking version are the non-blocking connect() implementation and handling of WANT_READ/WANT_WRITE retries for all SSL*() functions.

Timeouts in the tests needed to be increased because socket connect() and TLS SSL_connect() were not included in the timeout before. The tests don't run any slower, though. In fact, all platforms but Ubuntu 12.04 worked fine with the shorter timeouts.
2020-04-16 16:05:44 -04:00
David Steele
2260a7512a Use poll() instead of select() for monitoring socket read/write ready.
select() is a bit old-fashioned and cumbersome to use. Since the select() code needed to be modified to handle write ready this seems like a good time to upgrade to poll().

poll() has been around for a long time so there doesn't seem to be any need to provide a fallback to select().

Also change the error on timeout from FileReadError to ProtocolError. This works better for read vs. write and failure to poll() is indicative of a protocol error or unexpected EOF.
2020-04-16 15:02:33 -04:00
David Steele
e5e81d3839 Only limit backup copy size for WAL-logged files.
The prior behavior introduced in dcddf3a5 could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup since they are copied via tmp files and could change size during the backup.

In general it seems safer to limit this feature to WAL-logged files which will be reconstructed during recovery.
2020-04-16 14:48:16 -04:00
David Steele
0c07c5111f Remove empty subexpression from manifest regular expression.
MacOS was not happy about this though other platforms seemed to work fine.

This was likely a typo or a relic from refactoring.
2020-04-14 15:33:43 -04:00
David Steele
8fb0a8fd6e Remove incorrect Github id.
This was a copy-pasto from another contributor entry.
2020-04-14 15:28:43 -04:00
David Steele
9ffa2c618f Use SocketSession/TlsSession for test servers.
A session looks much the same whether it is initiated from the client or the server, so use the session objects to implement the TLS, HTTP, and S3 test servers.

For TLS, at least, there are some differences between client and server sessions so add a client/server type to SocketSession to determine how the session was initiated.

Aside from reducing code duplication, the main advantage is that the test server will now timeout rather than hanging indefinitely when less input that expected is received.
2020-04-14 15:22:49 -04:00
David Steele
71fb28bf3f Always throw error when OpenSSL returns SSL_ERROR_SYSCALL.
Previously an error was only thrown when errno was set but in practice this is usually not the case. This may have something to do with getting errno late but attempts to get it earlier have not been successful. It appears that errno usually gets cleared and spot research seems to indicate that other users have similar issues.

An error at this point indicates unexpected EOF so it seems better to just throw an error all the time and be consistent.

To test this properly our test server needs to call SSL_shutdown() except when the client expects this error.
2020-04-14 15:20:50 -04:00
David Steele
9f2d647bad Split session functionality of TlsClient out into TlsSession.
This abstraction allows the session code to be shared between the TLS client and (upcoming) server code.

Session management is no longer implemented in TlsClient so the HttpClient was updated to free and create sessions as needed. No test changes were required for HttpClient so the functionality should be unchanged.

Mechanical changes to the TLS tests were required to use TlsSession where appropriate rather than TlsClient. There should be no change in functionality other than how sessions are managed, i.e. using tlsClientOpen()/tlsSessionFree() rather than just tlsClientOpen().
2020-04-14 15:02:18 -04:00
David Steele
c9481bb95f Update contributing.xml to use Ubuntu 18.04.
Ubuntu 19.04 was EOL in January and has been removed from the repository servers.

It may be best to stick to LTS versions from now on.
2020-04-14 14:51:14 -04:00
David Steele
f03d1b5b7b Use __noreturn_ on error functions when coverage testing.
The errorInternalThrowSys*() functions were marked as returning during coverage testing even when they had no possibility to return, i.e. the error parameter was set to constant true. This meant the compiler would treat the functions as returning even when they would not.

Instead create completely separate functions for coverage to use for THROW_ON_SYS_ERROR*() that can return and leave the regular functions marked __noreturn__.
2020-04-14 11:43:50 -04:00
David Steele
b7d8d61526 Split session functionality of SocketClient out into SocketSession.
This abstraction allows the session code to be shared between the socket client and (upcoming) server code. There should no difference in how the code works -- only the organization has changed. Note that no changes to the tests were required.

This same abstraction will be required for TlsClient but that will be done in a separate commit because it requires test changes.
2020-04-13 16:59:02 -04:00
David Steele
9a751c2d98 Fix typo. 2020-04-10 12:56:28 -04:00
Cynthia Shang
63b9f46a34 Update contributing documentation and remove test/README.md.
When the Vagrant file was updated to use pgbackrest/ vs /backrest/ as the location for executing tests and building the documentation, parts of the contributing.xml (and hence the CONTRIBUTING.md) were not updated since some parts of the document are not actually executed when the CONTRIBUTING.md is built from contributing.xml: those parts that are executed were updated but those parts that are not executed were not.

This commit fixes the contributing.xml issue but also removes test/README.md as its contents were out of date and redundant given that they are covered in CONTRIBUTING.md.
2020-04-09 18:25:25 -04:00
David Steele
2ec379fa40 Add ideator to release note. 2020-04-07 11:10:58 -04:00
David Steele
5e55d58850 Simplify storage driver info and list functions.
The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent.

Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact.

Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs.

Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
David Steele
1aca2cc902 Move extern function comments to headers.
This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them).

Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation.

Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left.

Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc.

A few function prototypes were found that not longer had an implementation. These were removed, but there may be more.

The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
David Steele
789e364e6b Rename tcp-keep-alive option to sck-keep-alive.
This is really a socket option so the new name is clearer.

Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.
2020-04-01 15:44:51 -04:00
David Steele
5c6fb88bef TCP keep-alive options are configurable.
Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.

Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.

Also move configuration of TCP options before connect() as PostgreSQL does.
2020-03-31 18:13:11 -04:00
Adrian Vondendriesch
e1c72f6f97 Fix typos. 2020-03-28 17:48:57 -04:00
David Steele
fa86422371 Begin v2.26 development. 2020-03-26 17:20:58 -04:00
David Steele
fd3dca036b v2.25: LZ4 Compression Support
Features:

* Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari.)

Improvements:

* Improve performance of remote manifest build. (Suggested by Jens Wilke.)
* Fix detection of keepalive options on Linux. (Contributed by Marc Cousin.)
* Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin.)
* Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
* Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
2020-03-26 17:14:53 -04:00
David Steele
d242114dc0 Rough script to add tables to the user guide for performance tests.
This really needs to be worked on but for now at least preserve to code to be improved later.
2020-03-25 19:00:09 -04:00
David Steele
4c31a922a3 Remove backup after timeline switch from user guide.
It is no longer necessary (since 2.07, 34c63276) to do a backup manually after a timeline switch.
2020-03-25 12:17:20 -04:00
David Steele
98e5affa8a Free temp mem context periodically in storageRemoteInfoList().
Other storage*InfoList() functions do this but it was missed here.

memResize()/memFree() operations become more expensive as the mem context grows larger so freeing it periodically saves processing time.
2020-03-24 18:44:13 -04:00
David Steele
4c831d8e83 Use --clean-only for reproducible builds in contributing documentation.
If the work or result directories already contain data then the docs might be generated slightly differently.  Doing a clean ensures they will always produce the same output (provided the code does not change).
2020-03-22 13:50:31 -04:00
David Steele
d70ca2592f Suppress timestamps in contributing documentation.
Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 10:17:03 -04:00
David Steele
a180251011 Do variable replacements for cmd-extra in documentation. 2020-03-22 09:59:38 -04:00
David Steele
6a9fbccf76 Remove custom lcov build from contributing documentation.
Using the stock lcov no longer seems to be a problem, probably due to a better implementation in CoverageTest.pm.
2020-03-21 19:08:27 -04:00
David Steele
56fb399373 Build contributing documentation on Travis CI.
Building the contributing document has some special requirements because it runs Docker in Docker so the repo path must align on the host and all Docker containers.  Run `pgbackrest/doc/doc.pl` from within the home directory of the user that will do the doc build, e.g. `home/vagrant`.  If the repo is not located directly in the home directory, e.g. `/home/vagrant/pgbackrest`, then a symlink may be used, e.g. `ln -s /path/to/repo /home/vagrant/pgbackrest`.

Mount the repo in the Vagrantfile at /home/vagrant/pgbackrest but provide a link from the old location at /backrest to make the transition less painful.
2020-03-21 18:45:58 -04:00
David Steele
ee2e15bf55 Add missing package to contributing documentation. 2020-03-21 09:51:12 -04:00
David Steele
dcddf3a58b Limit backup file copy size to size reported at backup start.
If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data.

This also reduces the likelihood of seeing torn pages during the copy. Torn pages can still occur in the middle of the file, though, so they must be handled.
2020-03-19 13:16:05 -04:00
David Steele
e5bcc0c47e Clarifications of what limit == NULL or UINT64_MAX means. 2020-03-19 12:34:27 -04:00
Cynthia Shang
f538da0571 Updates to contributing documentation. 2020-03-18 17:30:36 -04:00
Cynthia Shang
2fa69af8da Add --dry-run option to the expire command.
Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.
2020-03-16 13:56:52 -04:00
David Steele
4cd060b7fe Generate src/build/aclocal.m4 automatically.
This file is required when macros from the autoconf archive are used in configure.ac
2020-03-14 12:48:08 -04:00
Marc Cousin
4b24a74afb Fix detection of keepalive options on Linux.
This code stanza was not being included on Linux platforms because of a missing header file.

Also update the order of operations and make the timeout calculations more sensible.
2020-03-12 11:38:50 -04:00
Marc Cousin
d6ef6c57c3 Add configure host detection to set standards flags correctly.
Linux and MacOS were tolerant of having all standards flags set but BSD was not.

Instead detect the host type and set standards flags as appropriate.
2020-03-12 10:37:19 -04:00
David Steele
fa1f63198d Add scripts required for host detection by src/configure.
Add to the src/build directory so they will be out of the way.

Also add instructions for updating the scripts to RELEASE.md.
2020-03-12 10:03:57 -04:00
David Steele
ccdf4bf3cc Add repo path env var to make release commands easier to run. 2020-03-12 09:43:29 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
d3c83453de Add repo-create, repo-get, repo-put, and repo-rm commands.
These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage.

All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
David Steele
5e1291a29f Rename ls command to repo-ls.
This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands.  Since the repo storage can be located lots of places having a common way to list it makes sense.

Prefix with repo- to make the scope of this command clear.

Update documentation to reflect this change.
2020-03-09 16:41:04 -04:00
David Steele
438b957f9c Add infrastructure for multiple compression type support.
Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.

Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).

Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.

Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
Cynthia Shang
089049ec56 Add sleep before/after retrieving timestamp in the user guide.
Adding a sleep before was necessary since only adding a sleep after did not always work. This helps to ensure the backup stop time for the previous backup does not equal time-recovery-timestamp. The sleep after allows enough time between the time retrieval and dropping important_table so PostgreSQL can consistently recover to before the table drop.

Note that these issues were caused by picking a timestamp too close to the restore command or a database operation, not due to any problem in backup selection of the restore command.
2020-02-28 14:30:39 -05:00
David Steele
dbf6255ab8 Remove compress/compress-level options from commands where unused.
These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 12:25:32 -05:00
David Steele
ee351682da Rename "gzip" to "gz".
"gz" was used as the extension but "gzip" was generally used for function and type naming.

With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code.  Since the extension is standard and we must use it, also use the extension for all naming.
2020-02-27 12:09:05 -05:00
David Steele
5afd950ed9 Improve performance of MEM_CONTEXT*() macros.
The prior code used TRY...CATCH blocks to cleanup mem contexts when an error occurred. This included freeing new mem contexts that were still being initialized when the error occurred and ensuring that the prior memory context was restored.

This worked fine in production but it involved a lot of setjmp()/longjmp() calls that resulted in longer compilation times and sluggish performance under valgrind, profiling, and coverage testing.

Instead maintain a stack of new contexts and context switches that can be used to do cleanup after an error. Normally, the stack is not used for this purpose and pushing/popping is a cheap operation. In the prior implementation most of the TRY...CATCH logic needed to be run even on success.

One bonus is that the binary is about 8% smaller after this change.  Another benefit is that new contexts *must* be explicitly freed/discarded or an error will occur.  See info/manifest.c for an example of where this is useful outside the standard macros.
2020-02-26 21:15:39 -05:00
Cynthia Shang
d68771a4a5 Fix incorrect lcov version in contributing guide. 2020-02-26 20:40:24 -05:00
David Steele
9e0dc83e87 Begin v2.25 development. 2020-02-25 17:18:25 -05:00
David Steele
495dec44f0 v2.24: Auto-Select Backup Set for Time Target
Bug Fixes:

* Prevent defunct processes in asynchronous archive commands. (Reviewed by Stephen Frost. Reported by Adam Brusselback, ejberdecia.)
* Error when archive-get/archive-push/restore are not run on a PostgreSQL host. (Reviewed by Stephen Frost. Reported by Jesper St John.)
* Read HTTP content to eof when size/encoding not specified. (Reviewed by Cynthia Shang. Reported by Christian ROUX.)
* Fix resume when the resumable backup was created by Perl. In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully. (Reported by Kacey Holston.)

Features:

* Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. (Contributed by Cynthia Shang.)

Improvements:

* Skip pg_internal.init temp file during backup. (Reviewed by Cynthia Shang. Suggested by Michael Paquier.)
* Add more validations to the manifest on backup. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Prevent lock-bot from adding comments to locked issues. (Suggested by Christoph Berg.)
2020-02-25 17:05:45 -05:00
David Steele
cc743f2e04 Skip pg_internal.init temp file during backup.
If PostgreSQL crashes it can leave behind a pg_internal.init temp file with the pid as the extension, as discussed in https://www.postgresql.org/message-id/flat/20200131045352.GB2631%40paquier.xyz#7700b9481ef5b0dd5f09cc410b4750f6.  On restart this file is not cleaned up so it can persist for the lifetime of the cluster or until another process with the same id happens to write pg_internal.init.

This is arguably a bug in PostgreSQL, but in any case it makes sense not to backup this file.
2020-02-21 11:51:39 -05:00
David Steele
ea0af890d8 Reclassify release note to documentation improvement. 2020-02-12 17:27:44 -07:00
David Steele
c6b89d74ec Add reviewer. 2020-02-12 17:20:21 -07:00
David Steele
6353e9428d Error when archive-get/archive-push/restore are not run on a PostgreSQL host.
This error was lost during the migration to C.  The error that occurred instead (generally an SSH auth error) was hard to debug.

Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands.  reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
David Steele
e2c304d473 Prevent defunct processes in asynchronous archive commands.
The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.

In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.

Also update the comment on chdir() to more accurately reflect what is happening.

Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).
2020-02-12 12:17:23 -07:00
David Steele
1be9e6854e Prevent lock-bot from adding comments to locked issues.
This will hopefully prevent users from getting notifications when an issue is locked.
2020-02-11 19:52:23 -07:00
David Steele
43936c58a8 Fix resume when the resumable backup was created by Perl.
In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully.
2020-02-11 19:44:06 -07:00
Mike Palmiotto
efff54490f Fix release note typo. 2020-02-04 21:19:21 -08:00
David Steele
296aec03be Update contributor name. 2020-01-31 07:50:03 -07:00
David Steele
0f8ec3e478 Read HTTP content to eof when size/encoding not specified.
Generally, the content-size or content-encoding headers will be used to specify how much content should be expected.

There is a special case where the server sends 'Connection:close' without the content headers and the content may be read up until eof.

This appears to be an atypical usage but it is required by the specification.
2020-01-30 14:51:26 -07:00
Cynthia Shang
856980ae99 Auto-select backup set on restore when time target is specified.
Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.

Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 14:38:05 -07:00
David Steele
0a845214a1 Fix typo. 2020-01-26 23:10:29 -07:00
David Steele
697150eaf8 Add more validations to the manifest on backup.
Validate that checksums exist for zero size files.  This means that the checksums for zero size files are explicitly set by backup even though they'll always be the same.  Also validate that zero length files have the correct checksum.

Validate that repo size is > 0 if size is > 0.  No matter what compression type is used a non-zero amount of data cannot be stored in zero bytes.
2020-01-26 23:07:07 -07:00
David Steele
bb45a80d46 Begin v2.24 development. 2020-01-26 22:47:53 -07:00
David Steele
2358d34485 v2.23: Bug Fix
Bug Fixes:

* Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)

Improvements:

* Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
* Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
2020-01-26 22:38:21 -07:00
David Steele
7ab07dc580 Validate checksums are set in the manifest on backup/restore.
This is a modest start but it addresses the specific issue that was caused by the bug fixed in 45ec694a.  This validation will produce an immediate error rather than erroring out partway through the restore.

More validations are planned but this is the most important one and seems safest for this release.
2020-01-26 21:58:59 -07:00
David Steele
45ec694af2 Fix missing files corrupting the manifest.
If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored.

The issue was that removing files from the manifest invalidated the pointers stored in the processing queues.  When a file was removed, all the pointers shifted to the next file in the list, causing a file to be unprocessed.  Since the unprocessed file was still in the manifest it would be saved with no checksum, causing a failure on restore.

When process-max was > 1 then the bug would often not express since the file had already been pulled from the queue and updates to the manifest are done by name rather than by pointer.
2020-01-26 13:19:13 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
1706c599bd Set encoding to UTF8 for Debian documentation containers.
This allows testing multi-byte encodings in PostgreSQL.
2020-01-21 18:37:43 -07:00
David Steele
382ddfd79d Begin v2.23 development. 2020-01-21 16:43:44 -07:00
David Steele
bae6e1c9e3 v2.22: Bug Fix
Bug Fixes:

* Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
2020-01-21 16:12:33 -07:00
David Steele
b89e6b7f69 Fix error in timeline conversion.
The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA.
2020-01-21 10:29:46 -07:00
David Steele
193b78e150 Remove -s from make instructions.
It's a good option but not required so let the user make the choice.

Also combine configure and make into a single line.
2020-01-15 13:40:17 -07:00
David Steele
9414ad2300 Begin v2.22 development. 2020-01-15 13:29:52 -07:00
David Steele
2c0ba0820d v2.21: C Migration Complete
Bug Fixes:

* Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
* Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.

Features:

* pgBackRest is now pure C.
* Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto.)
* Allow path-style URIs in S3 driver.

Improvements:

* The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
2020-01-15 13:21:52 -07:00
David Steele
8d3710b2fe Fix options being ignored by asynchronous commands.
The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc.

Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section.

Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
David Steele
fe263e87b1 Allow path-style URIs in S3 driver.
Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.

Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.
2020-01-12 11:31:06 -07:00
David Steele
61538f932c Parse dates in storageS3InfoList() and storageS3Info().
Previously dates were not being filled by these functions which was fine since dates were not used.

We plan to use dates for the ls command plus it makes sense for the driver to be complete since it will be used as an example.
2020-01-06 15:53:53 -07:00
David Steele
d2fb4f977c Add httpLastModifiedToTime() to parse HTTP last-modified header. 2020-01-06 15:24:49 -07:00
David Steele
a08298ce1b Add basic time management functions.
These are similar to what mktime() and strptime() do but they ignore the local system timezone which saves having to munge the TZ env variable to do time conversions.
2020-01-06 15:18:52 -07:00
David Steele
d41eea685a Change meaning of TEST_RESULT_STR() macro.
This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping.

TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration.

Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
David Steele
74c3842595 Remove errant tabs and fix spacing. 2019-12-19 16:25:46 -05:00
Mike Palmiotto
dc1e7ca22d Add pg-user option.
Specifies the database user name when connecting to PostgreSQL.

If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior.
2019-12-19 11:26:38 -05:00
David Steele
ac35dcac39 Add note about symlink required for contributing documentation to build. 2019-12-17 21:28:40 -05:00
Stephen Frost
3394a50170 Remove unused Perl library missed in f0ef73db. 2019-12-17 21:25:54 -05:00
David Steele
03849840b8 Fix handling of \ in filenames.
\ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading.

Use jsonFromStr() to properly quote and escape \.

Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.
2019-12-13 21:33:13 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
1f2ce45e6b The backup command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C except as noted below.

A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.

The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.

For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.

Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.

Resume will now work even if hardlink settings have been changed.

Reviewed by Cynthia Shang.
2019-12-13 17:14:26 -05:00
David Steele
94cb9540f1 Begin v2.21 development. 2019-12-12 09:05:10 -05:00
David Steele
39fc2b7ad6 v2.20: Bug Fixes
Bug Fixes:

* Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)
* Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.
* Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)
2019-12-12 08:20:21 -05:00
David Steele
0194a98671 Fix archive-push/archive-get when PGDATA is symlinked.
Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked.

If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
David Steele
8c840c28a6 Fix segfault on unexpected EOF in gzip decompression.
If the compressed stream terminated early then the decompression process would get a flush request (NULL input buffer) since the filter was not marked as done.  This could happen on a zero-length or truncated (i.e. invalid) compressed file.

Change the existing assertion to an error to catch this condition in production gracefully.
2019-12-11 08:48:46 -05:00
David Steele
61ad3a44c9 Note that next repo format should use UTC for backup labels. 2019-12-09 09:12:34 -05:00
David Steele
d3132dae26 Add functions for building new manifests.
New manifests are built before a backup is performed.

Reviewed by Cynthia Shang.
2019-12-08 18:43:47 -05:00
David Steele
50eb062e0e Fix reference list when backup.info is reconstructed in expire command.
Adding a manifest to backup.info was migrated to C in 4e4d1f41 but deduplication of the references was missed leading to a reference for every file being added to backup.info.

Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual.

This unlikely combination of events means this is probably not a problem in the field.
2019-11-28 09:34:19 -05:00
David Steele
a4b9440d35 Only install specific lcov version when required.
Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.

Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.

PostgreSQL minor version releases are also included since all containers have been rebuilt.
2019-11-22 19:25:49 -05:00
David Steele
a6fc0bf2ca Add contributor. 2019-11-21 08:15:37 -05:00
David Steele
5f03471445 Remove --force option from stanza-create documentation.
This should have been removed when the support for the option was removed in c7333190.

The option cannot be removed entirely because we don't want to error in the case where --force was specified but the stanza is valid.
2019-11-19 15:56:13 -05:00
David Fetter
20162ed3fe Fix typo. 2019-11-19 10:32:18 -05:00
David Steele
1db9e3b144 Remove *MP() macros variants.
Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.

-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
David Steele
6f65dde8d3 Begin v2.20 development. 2019-11-12 15:56:28 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
David Steele
10c8eeaf6c Fix handling of repeated HTTP headers.
When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior.

Reported by donicrosby.
2019-11-08 18:58:45 -05:00
David Steele
4317178633 Update MinIO to newest release.
We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.

In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.
2019-11-08 17:56:34 -05:00
David Steele
eca00e0be0 Add building a development environment to contributing documentation.
This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros.

Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI.  We'll address this in the future when we add contributing documentation to the website.
2019-11-07 13:11:01 -05:00
David Steele
b3e5d88304 Add markdown renderer for execute-list block. 2019-11-02 09:59:39 +01:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
59a4a0c1b1 Add a warning about jq rounding numbers to the documentation. 2019-10-11 13:14:47 -04:00
Cynthia Shang
d90b2724f8 JSON output from the info command is no longer pretty-printed.
Monitoring systems can more easily ingest the JSON without linefeeds.

External tools such as jq can be used to pretty-print if desired.
2019-10-11 12:56:03 -04:00
David Steele
e4408c7dd3 Refactor --pre option in documentation.
Mark all pre commands as skip so they won't be run again after the container is built.

Ensure that pre commands added to the container are run as the container user if they are not intended to run as root.
2019-10-11 12:28:47 -04:00
David Steele
642ce003c8 Don't autogenerate embedded libc code by default.
This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.

Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.

Suggested by Stephen Frost.
2019-10-11 11:32:51 -04:00
David Steele
e3d87ebace Fix mismatched timezone in expect test.
Also run the --vm-none tests in a non-UTC timezone to prevent regressions.

Suggested by Stephen Frost.
2019-10-10 19:43:42 -04:00
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
50d1d0a4e8 Move release note to correct section. 2019-10-10 09:28:50 -04:00
David Steele
47e0ba2502 Fix remote timeout in delta restore.
When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout.

Add keep-alives to prevent remote timeout.

Reported by James Sewell, Jens Wilke.
2019-10-10 09:27:46 -04:00
Cynthia Shang
82baf1e521 Fix typo. 2019-10-10 08:33:40 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
a4152a0ea1 Update release instructions with a reminder to install latex.
Latex is no longer installed by default in the Vagrantfile.
2019-10-08 12:29:42 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
Brad Nicholson
887ec97e07 Document maximum version for auto-stop option. 2019-10-03 06:58:52 -04:00
David Steele
008ec6b05c Begin v2.19 development. 2019-10-01 17:04:26 -04:00
David Steele
4629237d6a v2.18: PostgreSQL 12 Support
Features:

* PostgreSQL 12 support.
* Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Suggested by Stephen Frost, ejberdecia.)
* Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)

Improvements:

* The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Suggested by James Chanco Jr.)
* Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang.)
* Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang.)
2019-10-01 16:02:07 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
0a4122c77d Add reviewer. 2019-09-27 13:34:59 -04:00
Cynthia Shang
6355451870 Fix old option name in documentation. 2019-09-27 13:33:14 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
8979336819 Comment out unprivileged user documentation.
We haven't had the time to complete this documentation and it has suffered bit rot.

This prevents us from building the docs on PostgreSQL >= 11 so just comment it all out until it can be updated.
2019-09-26 16:02:39 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
David Steele
f2781bdad4 Allow documentation to be built without encryption.
This is mainly useful for testing, where an encrypted repo is a real nuisance.
2019-09-25 16:38:13 -04:00
David Steele
6af64df9b6 Thoughts on repository version 6. 2019-09-23 19:26:24 -04:00
David Steele
c969137021 Migrate backup manifest load/save to C.
The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes,
timestamps, etc.  A list of databases is also included for selective restore.

The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that
nothing is missing or corrupt.  It is also useful for reporting, e.g. size of backup, backup time, etc.

For now, migrate enough functionality to implement the restore command.

Reviewed by Cynthia Shang.
2019-09-23 13:50:46 -04:00
Cynthia Shang
8675699d08 Add FAQ for time-based Point-in-Time Recovery. 2019-09-16 14:09:53 -04:00
Cynthia Shang
49b421d372 Add documentation clarifications regarding standby repositories.
Clarify that placing the repository on a standby is not a robust configuration and recommend against it.
2019-09-16 13:56:25 -04:00
Cynthia Shang
09ef03b7ef Add FAQ regarding backup-standby. 2019-09-15 08:27:38 -04:00
Cynthia Shang
56bf9d0566 Update HINT messages to conform to new standard detailed in CODING.md. 2019-09-14 12:21:08 -04:00
David Steele
f57e119b25 Document the relationship between db-timeout and protocol-timeout.
Contributed by Cynthia Shang.
Suggested by James Chanco Jr.
2019-09-13 12:28:39 -04:00
David Steele
15d04ca19c Add recursion and json output to the ls command.
These features finally make the ls command practical.

Currently the JSON contains only name, type, and size.  We may add more fields in the future, but these seem like the minimum needed to be useful.
2019-09-12 16:29:50 -04:00
David Steele
b53305cc36 Update incorrect pipelining references to connection reuse.
Connection reuse and pipelining are not the same thing and should not have been conflated.

Update comments and release notes to reflect the correct usage.
2019-09-12 08:17:07 -04:00
David Steele
4d84820021 Improve performance of info file load/save.
Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load.

This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files.

Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum.

Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests.

Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
David Steele
7334f30c35 Add helper function for adding CipherBlock filters to groups.
Reviewed by Cynthia Shang.
2019-09-06 13:35:28 -04:00
David Steele
8df7d68c8d Fix sudo missed in "Build pgBackRest as an unprivileged user".
286a106a updated the documentation to build pgBackRest as an unprivileged user, but the wget command was missed.  This command is not actually run, just displayed, because the release is not yet available when the documentation is built.

Update the wget command to run as the local user.
2019-09-03 18:28:53 -04:00
David Steele
005684bf1f Begin v2.18 development. 2019-09-03 17:53:50 -04:00
David Steele
ce2bf29998 v2.17: C Migrations and Bug Fixes
Bug Fixes:

* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)

Improvements:

* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)

Documentation Bug Fixes:

* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)

Documentation Improvements:

* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
2019-09-03 16:39:32 -04:00
David Steele
0b5720c642 Fix yum.p.o package being installed when custom package specified.
The {[os-type-is-centos]} expression was missing parens which meant "and" expressions built on it would always evaluate true if the os-type was centos6.

Reported by Joe Ayers, John Harvey.
2019-09-03 14:34:49 -04:00
Josh Soref
4a88791a0a Fix typos in the release notes.
Contributed by Josh Soref.
2019-08-26 12:29:43 -04:00
Josh Soref
545ccfa878 Fix typos in the documentation.
Contributed by Josh Soref.
2019-08-26 12:26:00 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
01c2669b97 Fix exclusions for special files.
Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.

Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.

Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.

Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 07:47:54 -04:00
David Steele
c002a2ce2f Move info file checksum to the end of the file.
Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.

Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.

There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.

Reviewed by Cynthia Shang.
2019-08-21 19:45:48 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
David Steele
286a106ae4 Build pgBackRest as an unprivileged user.
pgBackRest was being built by root in the documentation which is definitely not best practice.

Instead build as the unprivileged default container user.  Sudo privileges are still required to install.

Suggested by Laurenz Albe.
2019-08-20 09:46:29 -04:00
David Steele
9eaeb33c88 Fix slow manifest build for very large quantities of tables/segments.
storagePosixInfoList() processed each directory in a single memory context.  If the directory contained hundreds of thousands of files processing became very slow due to the number of allocations.

Instead, reset the memory context every thousand files to minimize the number of allocations active at once, improving both speed and memory consumption.

Reported by Jens Wilke.
2019-08-19 21:36:01 -04:00
David Steele
41b6795a37 Create log directories/files with 0750/0640 mode.
The log directories/files were being created with a mix of modes depending on whether they were created in C or Perl.  In particular, the C code was creating log files with the execute bit set for the user and group which was just odd.

Standardize on 750/640 for both code paths.

Suggested by Damiano Albani.
2019-08-17 14:15:37 -04:00
Cynthia Shang
382ed92825 The start/stop commands are implemented entirely in C.
The Perl versions remain because they are still being used by the Perl stanza commands.  Once the stanza commands are migrated they can be removed.

Contributed by Cynthia Shang.
2019-08-09 15:17:18 -04:00
David Steele
efc62c9057 Begin v2.17 development. 2019-08-05 12:32:06 -04:00
David Steele
9e730c1bd6 v2.16: C Migrations and Bug Fixes
Bug Fixes:

* Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)
* Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)
* Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)

Features:

* Add repo-s3-port option for setting a non-standard S3 service port.

Improvements:

* The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)
* The check command is implemented partly in C. (Reviewed by Cynthia Shang.)
2019-08-05 12:03:04 -04:00
David Steele
3d3003e9ca The check command is implemented partly in C.
Implement switch WAL and archive check in C but leave the rest in Perl for now.

The main idea was to have some real integration tests for the new database code so the rest of the migration can wait.

Reviewed by Cynthia Shang.
2019-08-01 20:35:01 -04:00
David Steele
e4901d50d5 Add Db object to encapsulate PostgreSQL queries and commands.
Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check.

Add the dbGet() helper function to get Db objects easily.

Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together.

Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
David Steele
f9e1f3a798 Retry S3 RequestTimeTooSkewed errors instead of immediately terminating.
The cause of this error seems to be that a failed request takes so long that a subsequent retry at the http level uses outdated headers.

We're not sure if pgBackRest it to blame here (in one case a kernel downgrade fixed it, in another case an incorrect network driver was the problem) so add retries to hopefully deal with the issue if it is not too persistent.  If SSL_write() has long delays before reporting an error then this will obviously affect backup performance.

Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.
2019-08-01 14:28:30 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
d8ca0e5c5b Add Perl interface to C PgQuery object.
This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.
2019-07-25 17:05:39 -04:00
David Steele
415542b4a3 Add PostgreSQL query client.
This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts.

Testing is performed using a shim that can use scripted responses to test all aspects of the client code.  The shim will be very useful for testing backup scenarios on complex topologies.

Reviewed by Cynthia Shang.
2019-07-25 14:50:02 -04:00
David Steele
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
David Steele
3bdba4933d Fix incorrect handling of transfer-encoding response to HEAD request.
The HTTP server can use either content-length or transfer-encoding to indicate that there is content in the response.  HEAD requests do not include content but return all the same headers as GET.  In the HEAD case we were ignoring content-length but not transfer-encoding which led to unexpected eof errors on AWS S3.  Our test server, minio, uses content-length so this was not caught in integration testing.

Ignore all content for HEAD requests (no matter how it is reported) and add a unit test for transfer-encoding to prevent a regression.

Found by Pavel Suderevsky.
2019-07-17 16:49:42 -04:00
Cynthia Shang
6a89c1526e Revert a2dcdc07.
It is simpler to implement the required logic in stanza-delete rather than add complexity to this function.

Contributed by Cynthia Shang.
2019-07-10 12:04:25 -04:00
David Steele
a22a6dc08c Update contributor name. 2019-07-10 06:06:07 -04:00
Cynthia Shang
a2dcdc0711 Update lockStopTest() to optionally return a result rather than error.
Some commands (e.g. stanza-delete) would prefer to throw a customized error.

Contributed by Cynthia Shang.
2019-07-09 16:41:58 -04:00
David Steele
5e1ed2e8a5 Remove clang static analysis.
This analysis never produced anything but false positives (var might be NULL) but took over a minute per test run and added 600MB to the test container.
2019-07-05 18:34:15 -04:00
David Steele
9836578520 Remove perl critic and coverage.
No new Perl code is being developed, so these tools are just taking up time and making migrations to newer platforms harder.  There are only a few Perl tests remaining with full coverage so the coverage tool does not warn of loss of coverage in most cases.

Remove both tools and associated libraries.
2019-07-05 16:55:17 -04:00
David Steele
fc21013522 Fix scoping violations exposed by optimizations in gcc 9.
gcc < 9 makes all compound literals function scope, even though the C spec requires them to be invalid outside the current scope.  Since the compiler and valgrind were not enforcing this we had a few violations which caused problems in gcc >= 9.

Even though we are not quite ready to support gcc 9 officially, fix the scoping violations that currently exist in the codebase.

Reported by chrlange, Ned T. Crigler.
2019-07-05 16:25:28 -04:00
David Steele
c55009d0f9 Community yum package can be installed with --var=package=yum.
Like apt, the community yum package can now be installed instead of a user-specified package.
2019-06-27 14:39:11 -04:00
David Steele
020101b30b Update release notes to explicitly select release commit when tagging. 2019-06-27 09:38:40 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
David Steele
bd6c0941e9 Fix missing dash in site name.
Without this the project name and tagline just run together.
2019-06-25 17:27:19 -04:00
David Steele
466602387b Begin v2.16 development. 2019-06-25 08:42:20 -04:00