1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2820 Commits

Author SHA1 Message Date
Stephen Frost
b4fc1804a8
Minor updates for bzip2 compression after more review.
Update error types throw by bzip2 to be more consistent with gzip.

Update the bzip2 and gzip error default to be AssertError as that's the more common case in both, and add a 'break;' to the default clause -- we don't intend to be just falling through those case statements, even if the default is the last, we should be explicit about that.

Clean up some tabs that snuck in, rename a variable to be more clear, and add some comments.
2020-05-08 16:27:54 -04:00
David Steele
14369c1c3c Mark variables modified in TRY block as volatile.
It's important that the values in these variables are maintained even after an exception is thrown, so they must be marked volatile.

Found while testing on the ppc64le architecture.
2020-05-08 15:36:20 -04:00
Cynthia Shang
cdebfb09e0
Add time-based retention for full backups.
The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days.

The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 15:25:03 -04:00
David Steele
e873ad6da0 Update Minio version to 2020-05-06T23-23-25Z in tests/documentation.
This release fixes the issue we submitted regarding an unquoted eTag: https://github.com/minio/minio/issues/9517
2020-05-07 17:26:46 -04:00
David Steele
faabf1227d Update Fedora container to Fedora 32.
This allows unit testing on gcc 10.

Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.
2020-05-07 11:06:56 -04:00
David Steele
6646446d2a Use Z_STREAM_END to detect gz compression finish.
Checking for free space in the output buffer worked, but if the buffer was completely filled then deflate() would need to be called again, which was wasteful and a bit confusing for debugging.

Instead, use Z_STREAM_END to detect that compression is done.

This change was inspired by the bz2 implementation in a021c9fe since bz2 does not allow BZ2_bzCompress() to be called after BZ_STREAM_END is returned. That made it obvious that gz would prefer the same implementation, even if it is more tolerant. The documentation at https://www.zlib.net/manual.html agrees.
2020-05-07 10:22:22 -04:00
David Steele
f8509ab76c Don't allow sudo to disable core dumps in test containers.
Newer versions of sudo output this message to stderr when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:38:28 -04:00
David Steele
3a75589855 Don't allow sudo to disable core dumps in documentation.
Newer versions of sudo output this message to stdout when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:23:15 -04:00
David Steele
ad784e1997 Add libz-dev to required build packages in Debian documentation.
This is apparently not installed by default in Ubuntu 20.04 as it was in prior versions.
2020-05-07 07:12:42 -04:00
David Steele
12a5d8a155 Add arm64 to Travis-CI build matrix.
apt.postgresql.org will soon be providing packages for arm64 so it's important that we support it.

Testing on multiple architectures also helps expose potential issues in popular architectures. See 10a5182d for an example.
2020-05-06 19:11:28 -04:00
David Steele
6be5ea3388 Suppress Valgrind errors on a per-VM basis.
There are a number of Valgrind errors on Ubuntu 12.04 which do not happen on newer distro versions. However, suppressions for these errors have masked legitimate issues in subsequent code.

Instead, make suppressions VM specific so errors in other VMs are not masked.
2020-05-06 18:24:48 -04:00
David Steele
28967951ab Fix leak in TlsClient object.
sckClientOpen() is the most likely part of this code to error so move it up above SSL session creation to reduce the chance of a leak.
2020-05-06 18:17:50 -04:00
David Steele
e677929802 Fix leak in CipherBlock object.
EVP_CIPHER_CTX_cleanup() was being called instead of EVP_CIPHER_CTX_free() so most of the memory was being freed but not all of it.

This leak was masked by Valgrind suppressions which are only applicable to Ubuntu 12.04, which will be addressed in a future commit.
2020-05-06 18:09:11 -04:00
David Steele
10a5182d62 Simplify retry handling in tls/http/socket clients.
Travis-CI arm64 was not happy with this pattern, perhaps because connected was being reset after a longjmp() even though it should have stayed with its originally initialized value of false. In any case, tlsClientOpen() ended up returning NULL on error rather than throwing an exception.

The new pattern seems simpler and passes all tests unmodified, so even though the error was only seen in TlsClient it makes sense to propagate to the other clients.
2020-05-06 15:00:34 -04:00
David Steele
8aede3353c Always use 127.0.0.1 on TLS tests outside of containers.
Resolving localhost can vary based on the local network configuration so it is safer to just use a static IP.

This was found while testing on Travis-CI arm64.
2020-05-06 14:49:03 -04:00
David Steele
3fe6ad5047 Build branches with -cit suffix on Travis-CI.
This allows a branch to be targeted at only Travis-CI and not other CI services.
2020-05-06 10:23:42 -04:00
David Steele
4c6dbe17ea Remove redundant Cirrus-CI branch filter. 2020-05-06 09:12:50 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
98f30ef222 Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.
PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.
2020-05-05 15:00:13 -04:00
David Steele
99405cbb15 Replace booleans with enums in compressType parameters.
This was an oversight in 438b957f which added multiple compression type support. The booleans were interpreted as none and gz which works fine for the CompressType enum until the position of gz or none changes.
2020-05-05 13:23:36 -04:00
David Steele
d04c21ca83 Centralize String and Buffer constants in stringz.h.
It's not clear how useful single-character zero-terminated constants are or if we want propagate them through the code, but it at least makes sense to centralize the constants used by the Buffer and String objects.
2020-05-04 19:05:38 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
David Steele
1aaaa94253 Remove Ubuntu 19.04 container definition.
Ubuntu 19.04 is no longer supported.
2020-05-04 14:02:25 -04:00
David Steele
39f5f3a0b4 Remove PostgreSQL 9.4 for Fedora 30 dropped from yum.p.o. 2020-05-04 13:12:52 -04:00
David Steele
64a21920e2 Move S3 initialization in user guide quickstart.
The previous location was too late to allow --var=s3-all=y to work with --require=/repo-host, which depends on /quickstart/configure-archiving.

Since the section is not included in production documentation, the position is not very important to flow so just move it to where it works.
2020-05-03 18:42:33 -04:00
David Steele
ef93249922 Add contributor for 816ba924 and reclassify as a bug. 2020-05-01 17:32:31 -04:00
David Steele
816ba9244f Allow pg-path1 to be optional for synchronous archive-push.
If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control.

Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.
2020-05-01 10:30:35 -04:00
David Steele
1d45282b97 Add missing spaces between while keyword and condition.
Our convention is to have a space here but some were missed.
2020-05-01 09:31:50 -04:00
David Steele
28ab65df10 Remove unused struct member.
Perhaps this was intended to verify the WAL size but was never implemented.

Verifying the WAL size is probably a good idea so this member may be added back if the feature is implemented.
2020-05-01 09:08:37 -04:00
David Steele
22ba1f02ce Convert storagePosixNew() to storagePosixNewP().
An upcoming feature requires new parameters for storagePosixNew() and this causes a lot of churn because almost every test creates a Posix storage object. Some refactoring in the tests might reduce this duplication but storagePosixNew() is collecting a lot of parameters so converting to storagePosixNewP() makes sense in any case.

There are relatively few call sites in the core code but they still benefit from better readability after this change.
2020-04-30 11:01:38 -04:00
David Steele
baf8cb9068 Fix issue checking if file links are contained in path links.
There is no conflict if the path containing a file link is a parent path of a path link. The Perl code apparently had this right but the migration to C missed it.

Exclude this case when checking for link conflicts.
2020-04-30 10:47:09 -04:00
David Steele
ce55866714 Enforce non-null for most string options.
There have been a number of segfaults reported because a string option expected to be non-null was actually null. This is generally due to options that are expected to be set but are in fact optional.

Protect against this by creating cfgOptionStrNull() to get options that can be null, while changing cfgOptionStr() to always expect non-null. There are relatively few places where nulls are expected.

There is definitely a chance for breakage here as null options might currently be working in the field but will be caught by this new check. Hopefully introducing the check early in the release cycle will allow us to catch any issues.
2020-04-30 10:34:44 -04:00
David Steele
1158e7d645 Move command-line decompress check in common/compress test.
It makes sense to do this check right after the first compression so any issues are caught early.

Also, none of the current compression formats omit decompressCmd so make the test mandatory.
2020-04-29 13:55:05 -04:00
Cynthia Shang
c5241e5007 Expire WAL archive only when repo-retention-archive threshold is met.
Previously when retention-archive was set (either by the user or by default), archives prior to the archive-start of the oldest remaining full backup (after backup expiration occurred) would be expired even though the retention-archive threshold had not been met. For example, if there were 1 full backup remaining after backup expiration and the retention-archive was set to 2 and retention-archive-type=full, then archives prior to the archive-start of the remaining full backup would still be removed even though retention-archive required 2 full backups remaining before archives should be expired.

The thought was to keep the archive directory clean and since the full backup did not require prior archives, it was safe to delete them. However, this has caused problems for some users in the past (because they needed the WAL for other purposes) and with the new adhoc and time-based retention features, it was decided that the archives should remain until the threshold was met. The archives will eventually be removed and if having them causes space issues, the expire command and the retention-archive can always be run and adjusted.
2020-04-29 08:06:49 -04:00
David Steele
775e81a74d Refactor regular expression error handling per Coverity report.
Coverity was concerned that regExpError() might return and lead to an invalid reference of "this". This was unlikely since the function should never return but Coverity didn't know that. Also, a difference in error-handling logic at the two sites could cause the issue Coverity reported if they were to get out of sync.

Fix by refactoring out the core error function so that it is clear it will never return.
2020-04-28 15:55:54 -04:00
David Steele
e421cf9dd3 Fix possible NULL reference reported by Coverity.
this->input is set to NULL when the read input goes to EOF but it was possible that this->input could be used again in a subsequent loop, according to Coverity.

In fact this would really only be a problem if EOF suddenly went back to false, which is not allowed. However, checking this->input is cheaper than calling ioReadEofDriver() driver on each loop so this change makes sense as an optimization and it makes Coverity happy, too.
2020-04-28 15:04:34 -04:00
David Steele
f764953b70 Check that options are valid when requested.
If an option may not be valid for a command it should be checked with cfgOptionValid() or cfgOptionTest().

It appears this rule is followed pretty strictly since the only changes required were in unit tests.
2020-04-27 17:25:38 -04:00
Cynthia Shang
1c1a710460 Add --set option to the expire command.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 14:00:36 -04:00
Cynthia Shang
ad33f545d1 Move latest backup link functionality to backup/common module.
This function is needed for new expire features.
2020-04-27 13:17:30 -04:00
Cynthia Shang
483838233f The expire command now checks if a stop file is present.
This check has always been missing but it was not noticed because expire is not usually run independently of backup, which does the check correctly.
2020-04-27 10:57:19 -04:00
Cynthia Shang
a5b2630d7c Throw user-friendly error if expire is not run on repository host.
Running the expire command on a non-repository host threw an assertion because the command is not yet configured for remote execution.
2020-04-27 10:49:05 -04:00
David Steele
987df62ec0 Centralize enforcement of option constraints.
Each option type enforced its own constraints but there was a lot of duplication. Centralize the enforcement to remove the duplication.

Also convert the option type assert to a production error. This is unlikely to happen in production but the test is quite cheap so it can't hurt.

Finally, add a NULL check. Most option types can never be NULL.
2020-04-26 17:22:37 -04:00
David Steele
12069ff8f3 Add reviewer. 2020-04-24 08:45:54 -04:00
David Steele
0dd6629a2d Automatically add default user as reviewer.
Contributions looked like they had no reviewer when reviewed by the default user.
2020-04-24 08:40:54 -04:00
Stefan Fercot
e92eb709d6 Add backup/expire running status to the info command.
This is implemented by checking for a backup lock on the host where info is running so there are a few limitations:

* It is not currently possible to know which command is running: backup, expire, or stanza-*. The stanza commands are very unlikely to be running so it's pretty safe to guess backup/expire. Command information may be added to the lock file to improve the accuracy of the reported command.

* If the info command is run on a host that is not participating in the backup, e.g. a standby, then there will be no backup lock. This seems like a minor limitation since running info on the repo or primary host is preferred.
2020-04-24 08:00:00 -04:00
David Steele
2e6938fad9 Restore works when PGDATA is a link.
Make the restore clean process look more like manifest build, i.e. do cleanup of each target root directory outside the main cleanup callback. This means some code duplication but removes the logic handling "dot" paths.

Add tests for both restore and backup (which already worked but was not tested).
2020-04-21 17:55:36 -04:00
Roman Peshkurov
1eb2678938 Fix minor leak in TLS host verification.
sk_GENERAL_NAME_free() only freed the name stack, not the names in the stack. sk_GENERAL_NAME_pop_free() frees both.

Due to aggressive connection reuse this leak was unlikely to be very noticeable.
2020-04-21 17:20:42 -04:00
David Steele
6d11fe53f7 Update RHEL package location (again).
This changed (again) upstream so update the file paths.
2020-04-21 10:51:38 -04:00
David Steele
a999500d2c Remove XML source for test/README.md.
63b9f46a removed test/README.md but did not remove the XML source so it reappeared when the release documentation was built.
2020-04-20 16:21:53 -04:00
David Steele
3e2dc05817 Begin v2.27 development. 2020-04-20 15:58:06 -04:00