1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1642 Commits

Author SHA1 Message Date
David Steele
67d9d3350c Update recovery.conf in user guide for PostgreSQL >= 12.
The postgresql.auto.conf file was being used instead of recovery.conf, but there were still instances in the text that used recovery.conf. Update to postgresql.auto.conf for PostgreSQL >= 10 and change wording where needed.
2020-07-17 13:27:14 -04:00
David Steele
24d2c5b277
Remove real/all integration tests now covered by unit tests.
Remove all check and stanza-* tests except for the ones that are intended to succeed. The successful tests show that the queries run with expected results against each version of PG which should also validate queries for the failure tests in the unit tests.

Also remove the tests for --no-online backups since they don't require a database and are well tested in the unit tests.
2020-07-16 13:57:14 -04:00
David Steele
332f2fb7f5 Update PostgreSQL versions used to build user guides.
Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version.
2020-07-16 12:54:52 -04:00
David Steele
232d14993d Fix CentOS path for PostgreSQL >= 10 logs in user guide.
Debian worked since logs default to another location, but CentOS/RHEL on PostgreSQL >= 10 requires the new location.
2020-07-16 12:45:24 -04:00
Stefan Fercot
047d85c263
Automatically determine cipher passphrase in repo-get command.
The prior code was only able to use the main passphrase automatically and expected sub passphrases to be specified for each operation. This was fine for testing but hardly sufficient for a user-facing feature.

Update the code to determine which passphrase to use for any file in the repository and error when an invalid file or location is selected.

The repo-get command is still internal for now, but with this improvement it should be ready to be made public.
2020-07-16 12:24:03 -04:00
David Steele
50ff5d905e Update comment and parameter in HttpRequest. 2020-07-15 13:54:01 -04:00
David Steele
574f36c9d2 Rename httpRequest() to httpRequestResponse() and fix comment. 2020-07-14 15:14:41 -04:00
David Steele
620a8d17cf
Automatic retry for backup, restore, archive-get, and archive-push.
If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors.

Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
Stefan Fercot
d3dd32a031
Add expire-auto option.
This allows automatic expiration after a successful backup to be disabled.
2020-07-14 08:12:25 -04:00
David Steele
2f7823c627
Add shared access signature (SAS) authorization for Azure.
A shared access signature (SAS) provides granular, delegated access to resources in a storage account. This is often preferable to using a shared key which provides more access and is a greater security risk if compromised.
2020-07-09 14:46:48 -04:00
David Steele
dd9e14b628 Add pgLsnFromWalSegment().
Provides the reverse operation for pgLsnToWalSegment().
2020-07-08 12:25:39 -04:00
David Steele
682ac656f5
Fix restore --force acting like --force --delta.
This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue.
2020-07-06 15:03:24 -04:00
David Steele
3f4371d7a2 Azure support for repository storage.
Azure and Azure-compatible object stores can now be used for repository storage.

Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
David Steele
be16bf69a8 Remove internal commands from HTML and man command references.
Some of these commands will be made public in the future but for now their interfaces are not stable so they remain internal.
2020-06-29 15:07:17 -04:00
David Steele
96adf8e513 PostgreSQL 13 beta2 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-06-26 07:44:56 -04:00
David Steele
e46eeefada
Add review for ea04ec7b. 2020-06-26 06:34:21 -04:00
David Steele
ea04ec7b3f Disable query parallelism in PostgreSQL sessions used for backup control.
There is no need to have parallelism enabled in a backup control session. In particular, 9.6 marks pg_stop_backup() as parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker.
2020-06-25 08:02:48 -04:00
David Steele
c5892d1291
Asynchronous S3 multipart upload.
When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part.

Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response.

Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously).

There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory).

A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality.

Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
2020-06-24 13:44:00 -04:00
David Steele
fbff29957c
Inline strPtr() to increase profiling accuracy.
strPtr() is called more than any other function and during profiling (with or without optimization) it can end up using a disproportionate amount of the total runtime. Even though it is fast, the profiler has a minimum resolution for each function call so strPtr() will often end up towards the top of the list even though the real runtime is quite small.

Instead, inline strPtr() and indicate to gcc that it should be inlined even for non-optimized builds, since that's how profiles are usually generated.

To make strPtr() smaller require "this" to be non-NULL and add another function, strPtrNull(), to deal with the few cases where we need NULL handling.

As a bonus this makes the executable about 1% smaller even when compared to a prior optimized build which would inline some percentage of strPtr() calls.
2020-06-18 13:13:55 -04:00
David Steele
3d74ec1190
Use PostgreSQL instead of postmaster where appropriate.
Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.
2020-06-17 15:14:59 -04:00
David Steele
0680cfc8dc Rename most instances of master to primary in tests.
This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702).

Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 14:06:38 -04:00
David Steele
11c192f30e
Add hint when checksum delta is enabled after a timeline switch.
This warning is normal when restoring a backup or promoting a standby so add a hint to make that clear.
2020-06-16 13:20:01 -04:00
Cynthia Shang
1094a2d802
Update the PITR FAQ to clarify the default behavior. 2020-06-12 11:27:18 -04:00
Cynthia Shang
a60d4c939a
Update FAQ page for expiring a specific backup set.
The FAQ should have been updated with the addition of ad hoc expire in 1c1a7104.
2020-06-11 14:06:36 -04:00
David Steele
6fe60a2428
Improve behavior of the repo-ls command.
* Exclude linefeed when there is no output to avoid a blank line.
* Honor filter when adding . path or listing a single file.
2020-06-11 13:17:35 -04:00
David Steele
237ba54d20
Fix expression when recursion enabled in storageInfoListP().
Expressions only worked at the first level of recursion because the expression was also being applied to paths so the path had to match the filter in order to recurse.

This is not considered a bug since it does not affect any existing code paths, but it is required for the general-purpose repo-ls command.
2020-06-11 11:48:42 -04:00
David Steele
da4f15663b Improve error when pg1-path option missing for archive-get command.
The assert thrown was not as descriptive as a proper option missing error.
2020-06-10 11:41:08 -04:00
David Steele
9efbafc84c Fix incorrect example for repo-retention-full-type option. 2020-06-01 13:19:47 -04:00
David Steele
3b5f76b434
Improve handling of invalid HTTP response status.
A truncated HTTP response status could lead to an an unfriendly error message, which would be retried, but could be confusing if the error was persistent and required debugging.

Improve the error handling overall to catch more error cases explicitly and respond better to edge cases.

Also update the terminology in comments to align with the RFC. Variable and function names were not changed because a refactor is intended for HTTP response and it doesn't seem worth the additional code churn.
2020-05-27 15:13:55 -04:00
David Steele
d92d0513c0 Update install-sh to v1.16.2.
No version bump for some reason.

https://git.savannah.gnu.org/gitweb/?p=automake.git;a=commit;h=6774c9b219cd78445964cd694588cfc4df9b8316
2020-05-26 08:40:40 -04:00
David Steele
943b80e1a7 Begin v2.28 development. 2020-05-26 08:30:27 -04:00
David Steele
d8214e0d78 v2.27: Expiration Improvements and Compression Drivers
Bug Fixes:

* Fix issue checking if file links are contained in path links. (Reviewed by Cynthia Shang. Reported by Christophe Cavallié.)
* Allow pg-path1 to be optional for synchronous archive-push. (Reviewed by Cynthia Shang. Reported by Jerome Peng.)
* The expire command now checks if a stop file is present. (Fixed by Cynthia Shang. Reviewed by David Steele.)
* Handle missing reason phrase in HTTP response. (Reviewed by Cynthia Shang. Reported by Tenuun.)
* Increase buffer size for lz4 compression flush. (Reviewed by Cynthia Shang. Reported by Eric Radman.)
* Ignore pg-host* and repo-host* options for the remote command. (Reviewed by Cynthia Shang. Reported by Pavel Suderevsky.)
* Fix possibly missing pg1-* options for the remote command. (Reviewed by Cynthia Shang. Reported by Andrew L'Ecuyer.)

Features:

* Time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. (Contributed by Cynthia Shang, Pierre Ducroquet. Reviewed by David Steele.)
* Ad hoc backup expiration. Allow the user to remove a specified backup regardless of retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Zstandard compression support. Note that setting compress-type=zst will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* bzip2 compression support. Note that setting compress-type=bz2 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Contributed by Stephen Frost. Reviewed by David Steele, Cynthia Shang.)
* Add backup/expire running status to the info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)

Improvements:

* Expire WAL archive only when repo-retention-archive threshold is met. WAL prior to the first full backup was previously expired after the first full backup. Now it is preserved according to retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Add local MD5 implementation so S3 works when FIPS is enabled. (Reviewed by Cynthia Shang, Stephen Frost. Suggested by Brian Almeida, John Kelley.)
* PostgreSQL 13 beta1 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
* Reduce buffer-size default to 1MiB. (Reviewed by Stephen Frost.)
* Throw user-friendly error if expire is not run on repository host. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2020-05-26 08:11:50 -04:00
David Steele
20d8c76b6c
Ignore pg-host* and repo-host* options for the remote command.
The purpose of the remote command is to get access to local resources, so a remote should never start another remote. However, this could happen if there were host settings on the remote host, which ended badly with lock errors, loops, etc.

Add pg-local and repo-local options to indicate that the resource is local even if there are host settings.

Note that for the time being these options are internal and not intended for general usage. However, this is likely the direction needed to allow for more symmetric and manageable configurations.
2020-05-22 13:51:26 -04:00
David Steele
35ab61da70 Remove extra spaces.
These spaces crept in over time and then got copy-pasted all over the place.
2020-05-22 09:28:50 -04:00
David Steele
ae75ffc173
Fix possibly missing pg1-* options for the remote command.
Some pg1-* options are required by the remote so if they are not provided in the remote's configuration file then it may cause a configuration error, depending on the operation. This currently only applies to the pg1-path option.

This is still an issue for repo-* options but the same solution cannot be applied because some repo-* options are secure and cannot be passed on the command-line.
2020-05-21 16:09:23 -04:00
David Steele
ec7b7c5a3e
PostgreSQL 13 beta1 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-05-21 13:46:16 -04:00
David Steele
f15d6104d2
Add local MD5 implementation so S3 works when FIPS is enabled.
S3 requires the Content-MD5 header for many requests but MD5 is not available via OpenSSL when FIPS is enabled because it is considered to be insecure.

Even though our usage does not present any security risks a local M5 implementation is required to circumvent the over-broad FIPS restriction.

Vendorize the MD5 implementation found at https://openwall.info/wiki/people/solar/software/public-domain-source-code/md5 and add full coverage for the module in the common/crypto unit tests.
2020-05-20 14:56:13 -04:00
David Steele
ea9147e2e0 Reduce buffer-size default to 1MiB.
The prior default was determined by benchmarking the Perl code prior to the 1.0 release. In general buffer allocation was more expensive in Perl so large buffers gave the best performance. This was due to multiple buffer allocations for each filter in an IO operation.

The C code allocates fixed buffers for each IO operation so the cost for buffer allocation is lower than Perl. That being the case it made sense to benchmark the C code to determine the optimal buffer default.

The performance/storage tests were used to measure the performance of a variety of filters. 1GiB of data was processed by each filter 10 times and the results of the tests were averaged.

While most buffer sizes gave similar performance, 1MiB appeared to perform the best overall. Of course, different architectures are likely to yield different results but this seems like a sensible default. The buffer-size option may still need to be manually configured to give optimal results.

Raw test data for reference:

4MB buffer (prior default)

copy time 1807ms, avg time 180ms, avg throughput: 5942MB/s
md5 time 14200ms, avg time 1420ms, avg throughput: 756MB/s
sha1 time 11431ms, avg time 1143ms, avg throughput: 939MB/s
sha256 time 23463ms, avg time 2346ms, avg throughput: 457MB/s
gzip -6 time 381199ms, avg time 38119ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

1MB buffer (new default)

copy time 1760ms, avg time 176ms, avg throughput: 6100MB/s
md5 time 13739ms, avg time 1373ms, avg throughput: 781MB/s
sha1 time 11025ms, avg time 1102ms, avg throughput: 973MB/s
sha256 time 22539ms, avg time 2253ms, avg throughput: 476MB/s
gzip -6 time 372995ms, avg time 37299ms, avg throughput: 28MB/s
lz4 -1 time 15118ms, avg time 1511ms, avg throughput: 710MB/s

512K buffer

copy time 1782ms, avg time 178ms, avg throughput: 6025MB/s
md5 time 13724ms, avg time 1372ms, avg throughput: 782MB/s
sha1 time 10959ms, avg time 1095ms, avg throughput: 979MB/s
sha256 time 22982ms, avg time 2298ms, avg throughput: 467MB/s
gzip -6 time 378120ms, avg time 37812ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

256K buffer

copy time 1805ms, avg time 180ms, avg throughput: 5948MB/s
md5 time 13706ms, avg time 1370ms, avg throughput: 783MB/s
sha1 time 11074ms, avg time 1107ms, avg throughput: 969MB/s
sha256 time 22588ms, avg time 2258ms, avg throughput: 475MB/s
gzip -6 time 372645ms, avg time 37264ms, avg throughput: 28MB/s
lz4 -1 time 16346ms, avg time 1634ms, avg throughput: 656MB/s
2020-05-19 16:58:49 -04:00
David Steele
a3d9d9a387 Handle missing reason phrase in HTTP response.
Reason phrases (e.g. OK) are optional in HTTP 1.1 but the space after the status code is not. When the reason phrase was missing the required space was trimmed along with the trailing CR leading to a format error.

Rework the logic to preserve the space and allow empty reason phrases.

Found while testing against the Backblaze S3-compatible API.
2020-05-19 08:20:33 -04:00
David Steele
ac5d46dc50 Increase buffer size for lz4 compression flush.
Some lz4 versions between r131 and 1.7.5 did not return a sufficient buffer size from LZ4F_compressBound() to allow LZ4F_compressEnd() to complete reliably. While this issue was fixed in lz4 1.7.5 there are affected versions in supported distributions such as CentOS/RHEL 7.

Use one of the hacks suggested in https://github.com/lz4/lz4/issues/290 to increase the buffer size enough for LZ4F_compressEnd() to complete. This means that a slightly larger buffer size is required for all versions but it seems worth it to (hopefully) to fix the issue in all lz4 versions.
2020-05-16 18:25:31 -04:00
Magnus Hagander
b8a5c3ac6f
Fix incorrect command in reference documentation.
Also update process to command to be more consistent with the surrounding text.
2020-05-12 13:13:04 -04:00
Stephen Frost
b4fc1804a8
Minor updates for bzip2 compression after more review.
Update error types throw by bzip2 to be more consistent with gzip.

Update the bzip2 and gzip error default to be AssertError as that's the more common case in both, and add a 'break;' to the default clause -- we don't intend to be just falling through those case statements, even if the default is the last, we should be explicit about that.

Clean up some tabs that snuck in, rename a variable to be more clear, and add some comments.
2020-05-08 16:27:54 -04:00
Cynthia Shang
cdebfb09e0
Add time-based retention for full backups.
The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days.

The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 15:25:03 -04:00
David Steele
e873ad6da0 Update Minio version to 2020-05-06T23-23-25Z in tests/documentation.
This release fixes the issue we submitted regarding an unquoted eTag: https://github.com/minio/minio/issues/9517
2020-05-07 17:26:46 -04:00
David Steele
3a75589855 Don't allow sudo to disable core dumps in documentation.
Newer versions of sudo output this message to stdout when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:23:15 -04:00
David Steele
ad784e1997 Add libz-dev to required build packages in Debian documentation.
This is apparently not installed by default in Ubuntu 20.04 as it was in prior versions.
2020-05-07 07:12:42 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
David Steele
64a21920e2 Move S3 initialization in user guide quickstart.
The previous location was too late to allow --var=s3-all=y to work with --require=/repo-host, which depends on /quickstart/configure-archiving.

Since the section is not included in production documentation, the position is not very important to flow so just move it to where it works.
2020-05-03 18:42:33 -04:00
David Steele
ef93249922 Add contributor for 816ba924 and reclassify as a bug. 2020-05-01 17:32:31 -04:00
David Steele
816ba9244f Allow pg-path1 to be optional for synchronous archive-push.
If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control.

Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.
2020-05-01 10:30:35 -04:00
David Steele
baf8cb9068 Fix issue checking if file links are contained in path links.
There is no conflict if the path containing a file link is a parent path of a path link. The Perl code apparently had this right but the migration to C missed it.

Exclude this case when checking for link conflicts.
2020-04-30 10:47:09 -04:00
David Steele
ce55866714 Enforce non-null for most string options.
There have been a number of segfaults reported because a string option expected to be non-null was actually null. This is generally due to options that are expected to be set but are in fact optional.

Protect against this by creating cfgOptionStrNull() to get options that can be null, while changing cfgOptionStr() to always expect non-null. There are relatively few places where nulls are expected.

There is definitely a chance for breakage here as null options might currently be working in the field but will be caught by this new check. Hopefully introducing the check early in the release cycle will allow us to catch any issues.
2020-04-30 10:34:44 -04:00
Cynthia Shang
c5241e5007 Expire WAL archive only when repo-retention-archive threshold is met.
Previously when retention-archive was set (either by the user or by default), archives prior to the archive-start of the oldest remaining full backup (after backup expiration occurred) would be expired even though the retention-archive threshold had not been met. For example, if there were 1 full backup remaining after backup expiration and the retention-archive was set to 2 and retention-archive-type=full, then archives prior to the archive-start of the remaining full backup would still be removed even though retention-archive required 2 full backups remaining before archives should be expired.

The thought was to keep the archive directory clean and since the full backup did not require prior archives, it was safe to delete them. However, this has caused problems for some users in the past (because they needed the WAL for other purposes) and with the new adhoc and time-based retention features, it was decided that the archives should remain until the threshold was met. The archives will eventually be removed and if having them causes space issues, the expire command and the retention-archive can always be run and adjusted.
2020-04-29 08:06:49 -04:00
Cynthia Shang
1c1a710460 Add --set option to the expire command.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 14:00:36 -04:00
Cynthia Shang
483838233f The expire command now checks if a stop file is present.
This check has always been missing but it was not noticed because expire is not usually run independently of backup, which does the check correctly.
2020-04-27 10:57:19 -04:00
Cynthia Shang
a5b2630d7c Throw user-friendly error if expire is not run on repository host.
Running the expire command on a non-repository host threw an assertion because the command is not yet configured for remote execution.
2020-04-27 10:49:05 -04:00
David Steele
12069ff8f3 Add reviewer. 2020-04-24 08:45:54 -04:00
David Steele
0dd6629a2d Automatically add default user as reviewer.
Contributions looked like they had no reviewer when reviewed by the default user.
2020-04-24 08:40:54 -04:00
Stefan Fercot
e92eb709d6 Add backup/expire running status to the info command.
This is implemented by checking for a backup lock on the host where info is running so there are a few limitations:

* It is not currently possible to know which command is running: backup, expire, or stanza-*. The stanza commands are very unlikely to be running so it's pretty safe to guess backup/expire. Command information may be added to the lock file to improve the accuracy of the reported command.

* If the info command is run on a host that is not participating in the backup, e.g. a standby, then there will be no backup lock. This seems like a minor limitation since running info on the repo or primary host is preferred.
2020-04-24 08:00:00 -04:00
David Steele
a999500d2c Remove XML source for test/README.md.
63b9f46a removed test/README.md but did not remove the XML source so it reappeared when the release documentation was built.
2020-04-20 16:21:53 -04:00
David Steele
3e2dc05817 Begin v2.27 development. 2020-04-20 15:58:06 -04:00
David Steele
2352ecf7c0 v2.26: Non-blocking TLS
Bug Fixes:

* Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis.)

Improvements:

* Non-blocking TLS implementation. (Reviewed by Slava Moudry, Cynthia Shang, Stephen Frost.)
* Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)
* TCP keep-alive options are configurable. (Suggested by Marc Cousin.)
* Add io-timeout option.
2020-04-20 15:44:31 -04:00
Cynthia Shang
93e4fe0199 Specify that the io-timeout option is measured in seconds. 2020-04-20 13:11:34 -04:00
David Steele
5d25e508ae Add io-timeout option.
Timeout used for connections and read/write operations.

Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte.
2020-04-17 09:18:52 -04:00
David Steele
c88684e2bf Non-blocking TLS implementation.
The prior blocking implementation seemed to be prone to locking up on some (especially recent) kernel versions. Since we were unable to reproduce the issue in a development environment we can only speculate as to the cause, but there is a good chance that blocking sockets were the issue or contributed to the issue.

So move to a non-blocking implementation to hopefully clear up these issues. Testing in production environments that were prone to locking shows that the approach is promising and at the very least not a regression.

The main differences from the blocking version are the non-blocking connect() implementation and handling of WANT_READ/WANT_WRITE retries for all SSL*() functions.

Timeouts in the tests needed to be increased because socket connect() and TLS SSL_connect() were not included in the timeout before. The tests don't run any slower, though. In fact, all platforms but Ubuntu 12.04 worked fine with the shorter timeouts.
2020-04-16 16:05:44 -04:00
David Steele
2260a7512a Use poll() instead of select() for monitoring socket read/write ready.
select() is a bit old-fashioned and cumbersome to use. Since the select() code needed to be modified to handle write ready this seems like a good time to upgrade to poll().

poll() has been around for a long time so there doesn't seem to be any need to provide a fallback to select().

Also change the error on timeout from FileReadError to ProtocolError. This works better for read vs. write and failure to poll() is indicative of a protocol error or unexpected EOF.
2020-04-16 15:02:33 -04:00
David Steele
e5e81d3839 Only limit backup copy size for WAL-logged files.
The prior behavior introduced in dcddf3a5 could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup since they are copied via tmp files and could change size during the backup.

In general it seems safer to limit this feature to WAL-logged files which will be reconstructed during recovery.
2020-04-16 14:48:16 -04:00
David Steele
0c07c5111f Remove empty subexpression from manifest regular expression.
MacOS was not happy about this though other platforms seemed to work fine.

This was likely a typo or a relic from refactoring.
2020-04-14 15:33:43 -04:00
David Steele
8fb0a8fd6e Remove incorrect Github id.
This was a copy-pasto from another contributor entry.
2020-04-14 15:28:43 -04:00
David Steele
9ffa2c618f Use SocketSession/TlsSession for test servers.
A session looks much the same whether it is initiated from the client or the server, so use the session objects to implement the TLS, HTTP, and S3 test servers.

For TLS, at least, there are some differences between client and server sessions so add a client/server type to SocketSession to determine how the session was initiated.

Aside from reducing code duplication, the main advantage is that the test server will now timeout rather than hanging indefinitely when less input that expected is received.
2020-04-14 15:22:49 -04:00
David Steele
71fb28bf3f Always throw error when OpenSSL returns SSL_ERROR_SYSCALL.
Previously an error was only thrown when errno was set but in practice this is usually not the case. This may have something to do with getting errno late but attempts to get it earlier have not been successful. It appears that errno usually gets cleared and spot research seems to indicate that other users have similar issues.

An error at this point indicates unexpected EOF so it seems better to just throw an error all the time and be consistent.

To test this properly our test server needs to call SSL_shutdown() except when the client expects this error.
2020-04-14 15:20:50 -04:00
David Steele
9f2d647bad Split session functionality of TlsClient out into TlsSession.
This abstraction allows the session code to be shared between the TLS client and (upcoming) server code.

Session management is no longer implemented in TlsClient so the HttpClient was updated to free and create sessions as needed. No test changes were required for HttpClient so the functionality should be unchanged.

Mechanical changes to the TLS tests were required to use TlsSession where appropriate rather than TlsClient. There should be no change in functionality other than how sessions are managed, i.e. using tlsClientOpen()/tlsSessionFree() rather than just tlsClientOpen().
2020-04-14 15:02:18 -04:00
David Steele
c9481bb95f Update contributing.xml to use Ubuntu 18.04.
Ubuntu 19.04 was EOL in January and has been removed from the repository servers.

It may be best to stick to LTS versions from now on.
2020-04-14 14:51:14 -04:00
David Steele
f03d1b5b7b Use __noreturn_ on error functions when coverage testing.
The errorInternalThrowSys*() functions were marked as returning during coverage testing even when they had no possibility to return, i.e. the error parameter was set to constant true. This meant the compiler would treat the functions as returning even when they would not.

Instead create completely separate functions for coverage to use for THROW_ON_SYS_ERROR*() that can return and leave the regular functions marked __noreturn__.
2020-04-14 11:43:50 -04:00
David Steele
b7d8d61526 Split session functionality of SocketClient out into SocketSession.
This abstraction allows the session code to be shared between the socket client and (upcoming) server code. There should no difference in how the code works -- only the organization has changed. Note that no changes to the tests were required.

This same abstraction will be required for TlsClient but that will be done in a separate commit because it requires test changes.
2020-04-13 16:59:02 -04:00
David Steele
9a751c2d98 Fix typo. 2020-04-10 12:56:28 -04:00
Cynthia Shang
63b9f46a34 Update contributing documentation and remove test/README.md.
When the Vagrant file was updated to use pgbackrest/ vs /backrest/ as the location for executing tests and building the documentation, parts of the contributing.xml (and hence the CONTRIBUTING.md) were not updated since some parts of the document are not actually executed when the CONTRIBUTING.md is built from contributing.xml: those parts that are executed were updated but those parts that are not executed were not.

This commit fixes the contributing.xml issue but also removes test/README.md as its contents were out of date and redundant given that they are covered in CONTRIBUTING.md.
2020-04-09 18:25:25 -04:00
David Steele
2ec379fa40 Add ideator to release note. 2020-04-07 11:10:58 -04:00
David Steele
5e55d58850 Simplify storage driver info and list functions.
The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent.

Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact.

Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs.

Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
David Steele
1aca2cc902 Move extern function comments to headers.
This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them).

Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation.

Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left.

Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc.

A few function prototypes were found that not longer had an implementation. These were removed, but there may be more.

The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
David Steele
789e364e6b Rename tcp-keep-alive option to sck-keep-alive.
This is really a socket option so the new name is clearer.

Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.
2020-04-01 15:44:51 -04:00
David Steele
5c6fb88bef TCP keep-alive options are configurable.
Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.

Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.

Also move configuration of TCP options before connect() as PostgreSQL does.
2020-03-31 18:13:11 -04:00
Adrian Vondendriesch
e1c72f6f97 Fix typos. 2020-03-28 17:48:57 -04:00
David Steele
fa86422371 Begin v2.26 development. 2020-03-26 17:20:58 -04:00
David Steele
fd3dca036b v2.25: LZ4 Compression Support
Features:

* Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari.)

Improvements:

* Improve performance of remote manifest build. (Suggested by Jens Wilke.)
* Fix detection of keepalive options on Linux. (Contributed by Marc Cousin.)
* Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin.)
* Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
* Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
2020-03-26 17:14:53 -04:00
David Steele
d242114dc0 Rough script to add tables to the user guide for performance tests.
This really needs to be worked on but for now at least preserve to code to be improved later.
2020-03-25 19:00:09 -04:00
David Steele
4c31a922a3 Remove backup after timeline switch from user guide.
It is no longer necessary (since 2.07, 34c63276) to do a backup manually after a timeline switch.
2020-03-25 12:17:20 -04:00
David Steele
98e5affa8a Free temp mem context periodically in storageRemoteInfoList().
Other storage*InfoList() functions do this but it was missed here.

memResize()/memFree() operations become more expensive as the mem context grows larger so freeing it periodically saves processing time.
2020-03-24 18:44:13 -04:00
David Steele
4c831d8e83 Use --clean-only for reproducible builds in contributing documentation.
If the work or result directories already contain data then the docs might be generated slightly differently.  Doing a clean ensures they will always produce the same output (provided the code does not change).
2020-03-22 13:50:31 -04:00
David Steele
d70ca2592f Suppress timestamps in contributing documentation.
Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 10:17:03 -04:00
David Steele
a180251011 Do variable replacements for cmd-extra in documentation. 2020-03-22 09:59:38 -04:00
David Steele
6a9fbccf76 Remove custom lcov build from contributing documentation.
Using the stock lcov no longer seems to be a problem, probably due to a better implementation in CoverageTest.pm.
2020-03-21 19:08:27 -04:00
David Steele
56fb399373 Build contributing documentation on Travis CI.
Building the contributing document has some special requirements because it runs Docker in Docker so the repo path must align on the host and all Docker containers.  Run `pgbackrest/doc/doc.pl` from within the home directory of the user that will do the doc build, e.g. `home/vagrant`.  If the repo is not located directly in the home directory, e.g. `/home/vagrant/pgbackrest`, then a symlink may be used, e.g. `ln -s /path/to/repo /home/vagrant/pgbackrest`.

Mount the repo in the Vagrantfile at /home/vagrant/pgbackrest but provide a link from the old location at /backrest to make the transition less painful.
2020-03-21 18:45:58 -04:00
David Steele
ee2e15bf55 Add missing package to contributing documentation. 2020-03-21 09:51:12 -04:00
David Steele
dcddf3a58b Limit backup file copy size to size reported at backup start.
If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data.

This also reduces the likelihood of seeing torn pages during the copy. Torn pages can still occur in the middle of the file, though, so they must be handled.
2020-03-19 13:16:05 -04:00
David Steele
e5bcc0c47e Clarifications of what limit == NULL or UINT64_MAX means. 2020-03-19 12:34:27 -04:00
Cynthia Shang
f538da0571 Updates to contributing documentation. 2020-03-18 17:30:36 -04:00
Cynthia Shang
2fa69af8da Add --dry-run option to the expire command.
Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.
2020-03-16 13:56:52 -04:00
David Steele
4cd060b7fe Generate src/build/aclocal.m4 automatically.
This file is required when macros from the autoconf archive are used in configure.ac
2020-03-14 12:48:08 -04:00
Marc Cousin
4b24a74afb Fix detection of keepalive options on Linux.
This code stanza was not being included on Linux platforms because of a missing header file.

Also update the order of operations and make the timeout calculations more sensible.
2020-03-12 11:38:50 -04:00
Marc Cousin
d6ef6c57c3 Add configure host detection to set standards flags correctly.
Linux and MacOS were tolerant of having all standards flags set but BSD was not.

Instead detect the host type and set standards flags as appropriate.
2020-03-12 10:37:19 -04:00
David Steele
fa1f63198d Add scripts required for host detection by src/configure.
Add to the src/build directory so they will be out of the way.

Also add instructions for updating the scripts to RELEASE.md.
2020-03-12 10:03:57 -04:00
David Steele
ccdf4bf3cc Add repo path env var to make release commands easier to run. 2020-03-12 09:43:29 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
d3c83453de Add repo-create, repo-get, repo-put, and repo-rm commands.
These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage.

All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
David Steele
5e1291a29f Rename ls command to repo-ls.
This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands.  Since the repo storage can be located lots of places having a common way to list it makes sense.

Prefix with repo- to make the scope of this command clear.

Update documentation to reflect this change.
2020-03-09 16:41:04 -04:00
David Steele
438b957f9c Add infrastructure for multiple compression type support.
Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.

Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).

Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.

Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
Cynthia Shang
089049ec56 Add sleep before/after retrieving timestamp in the user guide.
Adding a sleep before was necessary since only adding a sleep after did not always work. This helps to ensure the backup stop time for the previous backup does not equal time-recovery-timestamp. The sleep after allows enough time between the time retrieval and dropping important_table so PostgreSQL can consistently recover to before the table drop.

Note that these issues were caused by picking a timestamp too close to the restore command or a database operation, not due to any problem in backup selection of the restore command.
2020-02-28 14:30:39 -05:00
David Steele
dbf6255ab8 Remove compress/compress-level options from commands where unused.
These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 12:25:32 -05:00
David Steele
ee351682da Rename "gzip" to "gz".
"gz" was used as the extension but "gzip" was generally used for function and type naming.

With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code.  Since the extension is standard and we must use it, also use the extension for all naming.
2020-02-27 12:09:05 -05:00
David Steele
5afd950ed9 Improve performance of MEM_CONTEXT*() macros.
The prior code used TRY...CATCH blocks to cleanup mem contexts when an error occurred. This included freeing new mem contexts that were still being initialized when the error occurred and ensuring that the prior memory context was restored.

This worked fine in production but it involved a lot of setjmp()/longjmp() calls that resulted in longer compilation times and sluggish performance under valgrind, profiling, and coverage testing.

Instead maintain a stack of new contexts and context switches that can be used to do cleanup after an error. Normally, the stack is not used for this purpose and pushing/popping is a cheap operation. In the prior implementation most of the TRY...CATCH logic needed to be run even on success.

One bonus is that the binary is about 8% smaller after this change.  Another benefit is that new contexts *must* be explicitly freed/discarded or an error will occur.  See info/manifest.c for an example of where this is useful outside the standard macros.
2020-02-26 21:15:39 -05:00
Cynthia Shang
d68771a4a5 Fix incorrect lcov version in contributing guide. 2020-02-26 20:40:24 -05:00
David Steele
9e0dc83e87 Begin v2.25 development. 2020-02-25 17:18:25 -05:00
David Steele
495dec44f0 v2.24: Auto-Select Backup Set for Time Target
Bug Fixes:

* Prevent defunct processes in asynchronous archive commands. (Reviewed by Stephen Frost. Reported by Adam Brusselback, ejberdecia.)
* Error when archive-get/archive-push/restore are not run on a PostgreSQL host. (Reviewed by Stephen Frost. Reported by Jesper St John.)
* Read HTTP content to eof when size/encoding not specified. (Reviewed by Cynthia Shang. Reported by Christian ROUX.)
* Fix resume when the resumable backup was created by Perl. In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully. (Reported by Kacey Holston.)

Features:

* Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. (Contributed by Cynthia Shang.)

Improvements:

* Skip pg_internal.init temp file during backup. (Reviewed by Cynthia Shang. Suggested by Michael Paquier.)
* Add more validations to the manifest on backup. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Prevent lock-bot from adding comments to locked issues. (Suggested by Christoph Berg.)
2020-02-25 17:05:45 -05:00
David Steele
cc743f2e04 Skip pg_internal.init temp file during backup.
If PostgreSQL crashes it can leave behind a pg_internal.init temp file with the pid as the extension, as discussed in https://www.postgresql.org/message-id/flat/20200131045352.GB2631%40paquier.xyz#7700b9481ef5b0dd5f09cc410b4750f6.  On restart this file is not cleaned up so it can persist for the lifetime of the cluster or until another process with the same id happens to write pg_internal.init.

This is arguably a bug in PostgreSQL, but in any case it makes sense not to backup this file.
2020-02-21 11:51:39 -05:00
David Steele
ea0af890d8 Reclassify release note to documentation improvement. 2020-02-12 17:27:44 -07:00
David Steele
c6b89d74ec Add reviewer. 2020-02-12 17:20:21 -07:00
David Steele
6353e9428d Error when archive-get/archive-push/restore are not run on a PostgreSQL host.
This error was lost during the migration to C.  The error that occurred instead (generally an SSH auth error) was hard to debug.

Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands.  reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
David Steele
e2c304d473 Prevent defunct processes in asynchronous archive commands.
The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.

In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.

Also update the comment on chdir() to more accurately reflect what is happening.

Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).
2020-02-12 12:17:23 -07:00
David Steele
1be9e6854e Prevent lock-bot from adding comments to locked issues.
This will hopefully prevent users from getting notifications when an issue is locked.
2020-02-11 19:52:23 -07:00
David Steele
43936c58a8 Fix resume when the resumable backup was created by Perl.
In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully.
2020-02-11 19:44:06 -07:00
Mike Palmiotto
efff54490f Fix release note typo. 2020-02-04 21:19:21 -08:00
David Steele
296aec03be Update contributor name. 2020-01-31 07:50:03 -07:00
David Steele
0f8ec3e478 Read HTTP content to eof when size/encoding not specified.
Generally, the content-size or content-encoding headers will be used to specify how much content should be expected.

There is a special case where the server sends 'Connection:close' without the content headers and the content may be read up until eof.

This appears to be an atypical usage but it is required by the specification.
2020-01-30 14:51:26 -07:00
Cynthia Shang
856980ae99 Auto-select backup set on restore when time target is specified.
Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.

Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 14:38:05 -07:00
David Steele
0a845214a1 Fix typo. 2020-01-26 23:10:29 -07:00
David Steele
697150eaf8 Add more validations to the manifest on backup.
Validate that checksums exist for zero size files.  This means that the checksums for zero size files are explicitly set by backup even though they'll always be the same.  Also validate that zero length files have the correct checksum.

Validate that repo size is > 0 if size is > 0.  No matter what compression type is used a non-zero amount of data cannot be stored in zero bytes.
2020-01-26 23:07:07 -07:00
David Steele
bb45a80d46 Begin v2.24 development. 2020-01-26 22:47:53 -07:00
David Steele
2358d34485 v2.23: Bug Fix
Bug Fixes:

* Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)

Improvements:

* Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
* Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
2020-01-26 22:38:21 -07:00
David Steele
7ab07dc580 Validate checksums are set in the manifest on backup/restore.
This is a modest start but it addresses the specific issue that was caused by the bug fixed in 45ec694a.  This validation will produce an immediate error rather than erroring out partway through the restore.

More validations are planned but this is the most important one and seems safest for this release.
2020-01-26 21:58:59 -07:00
David Steele
45ec694af2 Fix missing files corrupting the manifest.
If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored.

The issue was that removing files from the manifest invalidated the pointers stored in the processing queues.  When a file was removed, all the pointers shifted to the next file in the list, causing a file to be unprocessed.  Since the unprocessed file was still in the manifest it would be saved with no checksum, causing a failure on restore.

When process-max was > 1 then the bug would often not express since the file had already been pulled from the queue and updates to the manifest are done by name rather than by pointer.
2020-01-26 13:19:13 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
1706c599bd Set encoding to UTF8 for Debian documentation containers.
This allows testing multi-byte encodings in PostgreSQL.
2020-01-21 18:37:43 -07:00
David Steele
382ddfd79d Begin v2.23 development. 2020-01-21 16:43:44 -07:00
David Steele
bae6e1c9e3 v2.22: Bug Fix
Bug Fixes:

* Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
2020-01-21 16:12:33 -07:00
David Steele
b89e6b7f69 Fix error in timeline conversion.
The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA.
2020-01-21 10:29:46 -07:00
David Steele
193b78e150 Remove -s from make instructions.
It's a good option but not required so let the user make the choice.

Also combine configure and make into a single line.
2020-01-15 13:40:17 -07:00
David Steele
9414ad2300 Begin v2.22 development. 2020-01-15 13:29:52 -07:00
David Steele
2c0ba0820d v2.21: C Migration Complete
Bug Fixes:

* Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
* Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.

Features:

* pgBackRest is now pure C.
* Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto.)
* Allow path-style URIs in S3 driver.

Improvements:

* The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
2020-01-15 13:21:52 -07:00
David Steele
8d3710b2fe Fix options being ignored by asynchronous commands.
The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc.

Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section.

Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
David Steele
fe263e87b1 Allow path-style URIs in S3 driver.
Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.

Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.
2020-01-12 11:31:06 -07:00
David Steele
61538f932c Parse dates in storageS3InfoList() and storageS3Info().
Previously dates were not being filled by these functions which was fine since dates were not used.

We plan to use dates for the ls command plus it makes sense for the driver to be complete since it will be used as an example.
2020-01-06 15:53:53 -07:00
David Steele
d2fb4f977c Add httpLastModifiedToTime() to parse HTTP last-modified header. 2020-01-06 15:24:49 -07:00
David Steele
a08298ce1b Add basic time management functions.
These are similar to what mktime() and strptime() do but they ignore the local system timezone which saves having to munge the TZ env variable to do time conversions.
2020-01-06 15:18:52 -07:00
David Steele
d41eea685a Change meaning of TEST_RESULT_STR() macro.
This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping.

TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration.

Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00