1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1095 Commits

Author SHA1 Message Date
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
David Steele
286a106ae4 Build pgBackRest as an unprivileged user.
pgBackRest was being built by root in the documentation which is definitely not best practice.

Instead build as the unprivileged default container user.  Sudo privileges are still required to install.

Suggested by Laurenz Albe.
2019-08-20 09:46:29 -04:00
David Steele
9eaeb33c88 Fix slow manifest build for very large quantities of tables/segments.
storagePosixInfoList() processed each directory in a single memory context.  If the directory contained hundreds of thousands of files processing became very slow due to the number of allocations.

Instead, reset the memory context every thousand files to minimize the number of allocations active at once, improving both speed and memory consumption.

Reported by Jens Wilke.
2019-08-19 21:36:01 -04:00
David Steele
41b6795a37 Create log directories/files with 0750/0640 mode.
The log directories/files were being created with a mix of modes depending on whether they were created in C or Perl.  In particular, the C code was creating log files with the execute bit set for the user and group which was just odd.

Standardize on 750/640 for both code paths.

Suggested by Damiano Albani.
2019-08-17 14:15:37 -04:00
Cynthia Shang
382ed92825 The start/stop commands are implemented entirely in C.
The Perl versions remain because they are still being used by the Perl stanza commands.  Once the stanza commands are migrated they can be removed.

Contributed by Cynthia Shang.
2019-08-09 15:17:18 -04:00
David Steele
efc62c9057 Begin v2.17 development. 2019-08-05 12:32:06 -04:00
David Steele
9e730c1bd6 v2.16: C Migrations and Bug Fixes
Bug Fixes:

* Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)
* Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)
* Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)

Features:

* Add repo-s3-port option for setting a non-standard S3 service port.

Improvements:

* The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)
* The check command is implemented partly in C. (Reviewed by Cynthia Shang.)
2019-08-05 12:03:04 -04:00
David Steele
3d3003e9ca The check command is implemented partly in C.
Implement switch WAL and archive check in C but leave the rest in Perl for now.

The main idea was to have some real integration tests for the new database code so the rest of the migration can wait.

Reviewed by Cynthia Shang.
2019-08-01 20:35:01 -04:00
David Steele
e4901d50d5 Add Db object to encapsulate PostgreSQL queries and commands.
Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check.

Add the dbGet() helper function to get Db objects easily.

Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together.

Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
David Steele
f9e1f3a798 Retry S3 RequestTimeTooSkewed errors instead of immediately terminating.
The cause of this error seems to be that a failed request takes so long that a subsequent retry at the http level uses outdated headers.

We're not sure if pgBackRest it to blame here (in one case a kernel downgrade fixed it, in another case an incorrect network driver was the problem) so add retries to hopefully deal with the issue if it is not too persistent.  If SSL_write() has long delays before reporting an error then this will obviously affect backup performance.

Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.
2019-08-01 14:28:30 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
415542b4a3 Add PostgreSQL query client.
This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts.

Testing is performed using a shim that can use scripted responses to test all aspects of the client code.  The shim will be very useful for testing backup scenarios on complex topologies.

Reviewed by Cynthia Shang.
2019-07-25 14:50:02 -04:00
David Steele
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
David Steele
3bdba4933d Fix incorrect handling of transfer-encoding response to HEAD request.
The HTTP server can use either content-length or transfer-encoding to indicate that there is content in the response.  HEAD requests do not include content but return all the same headers as GET.  In the HEAD case we were ignoring content-length but not transfer-encoding which led to unexpected eof errors on AWS S3.  Our test server, minio, uses content-length so this was not caught in integration testing.

Ignore all content for HEAD requests (no matter how it is reported) and add a unit test for transfer-encoding to prevent a regression.

Found by Pavel Suderevsky.
2019-07-17 16:49:42 -04:00
Cynthia Shang
6a89c1526e Revert a2dcdc07.
It is simpler to implement the required logic in stanza-delete rather than add complexity to this function.

Contributed by Cynthia Shang.
2019-07-10 12:04:25 -04:00
David Steele
a22a6dc08c Update contributor name. 2019-07-10 06:06:07 -04:00
Cynthia Shang
a2dcdc0711 Update lockStopTest() to optionally return a result rather than error.
Some commands (e.g. stanza-delete) would prefer to throw a customized error.

Contributed by Cynthia Shang.
2019-07-09 16:41:58 -04:00
David Steele
fc21013522 Fix scoping violations exposed by optimizations in gcc 9.
gcc < 9 makes all compound literals function scope, even though the C spec requires them to be invalid outside the current scope.  Since the compiler and valgrind were not enforcing this we had a few violations which caused problems in gcc >= 9.

Even though we are not quite ready to support gcc 9 officially, fix the scoping violations that currently exist in the codebase.

Reported by chrlange, Ned T. Crigler.
2019-07-05 16:25:28 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
David Steele
466602387b Begin v2.16 development. 2019-06-25 08:42:20 -04:00
David Steele
6650d8144c v2.15: C Implementation of Expire
Bug Fixes:

* Fix archive retention expiring too aggressively. (Fixed by Cynthia Shang. Reported by Mohamad El-Rifai.)

Improvements:

* The expire command is implemented entirely in C. (Contributed by Cynthia Shang.)
* The local command for restore is implemented entirely in C.
* Remove hard-coded PostgreSQL user so $PGUSER works. (Suggested by Julian Zhang, Janis Puris.)
* Honor configure --prefix option. (Suggested by Daniel Westermann.)
* Rename repo-s3-verify-ssl option to repo-s3-verify-tls. The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure). The old name will continue to be accepted.

Documentation Improvements:

* Add FAQ to the documentation. (Contributed by Cynthia Shang.)
* Use wal_level=replica in the documentation for PostgreSQL ≥ 9.6. (Suggested by Patrick McLaughlin.)
2019-06-25 08:29:06 -04:00
David Steele
c22e10e4a9 Honor configure --prefix option.
The --prefix option was entirely ignored and DESTDIR was a combination of DESTDIR and bindir.

Bring both in line with recommendations for autoconf and make as specified in https://www.gnu.org/software/make/manual/html_node/Directory-Variables.html and https://www.gnu.org/prep/standards/html_node/DESTDIR.html.

Suggested by Daniel Westermann.
2019-06-24 15:42:33 -04:00
Cynthia Shang
62715ebf2d Fix archive retention expiring too aggressively.
The problem expressed when repo1-archive-retention-type was set to diff.  In this case repo1-archive-retention ended up being effectively equal to one, which meant PITR recovery was only possible from the last backup.  WAL required for consistency was still preserved for all backups.

This issue is not present in the C migration committed at 434cd832, which was written before this bug was reported.  Even so, we wanted to note this issue in the release notes in case any other users have been affected.

Fixed by Cynthia Shang.
Reported by Mohamad El-Rifai.
2019-06-19 17:49:38 -04:00
Cynthia Shang
e2d791394a Add FAQ to the documentation.
Contributed by Cynthia Shang.
2019-06-18 18:42:47 -04:00
David Steele
434cd83285 The expire command is implemented entirely in C.
This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests.

Along the way at least one bug was fixed, see issue #748.

Contributed by Cynthia Shang.
2019-06-18 15:19:20 -04:00
David Steele
0efdf2576f Remove hard-coded PostgreSQL user so $PGUSER works.
The PostgreSQL user was hard-coded to the OS user which libpq will automatically use if $PGUSER is not set, so this code was redundant and prevented $PGUSER from working when set.

Suggested by Julian Zhang, Janis Puris.
2019-06-18 07:35:34 -04:00
Cynthia Shang
c64c9c0590 Add backup management functions to InfoBackup.
Allow current backups to be listed and deleted.

Also expose some constants required by expire and stanza-* commands.

Contributed by Cynthia Shang.
2019-06-17 06:59:06 -04:00
Cynthia Shang
44bafc127d Rename info*New() functions to info*NewLoad().
These names more accurately reflect what the functions do and follow the convention started in Info and InfoPg.

Also remove the ignoreMissing parameter since it was never used.

Contributed by Cynthia Shang.
2019-06-17 06:47:15 -04:00
David Steele
d7bd0c58cd Use wal_level=replica in the documentation for PostgreSQL >= 9.6.
The documentation was using wal_level=hot_standby which is a deprecated setting.

Also remove the reference to wal_level=archive since it is no longer supported and is not recommended for older versions.

Suggested by Patrick McLaughlin.
2019-06-05 07:27:24 -04:00
David Steele
ec9622cde8 Use the git log to ease release note management.
The release notes are generally a direct reflection of the git log.  So, ease the burden of maintaining the release notes by using the git log to determine what needs to be added.

Currently only non-dev items are required to be matched to a git commit but the goal is to account for all commits.

The git history cache is generated from the git log but can be modified to correct typos and match the release notes as they evolve.  The commit hash is used to identify commits that have already been added to the cache.

There's plenty more to do here.  For instance, links to the commits for each release item should be added to the release notes.
2019-05-22 18:54:49 -04:00
David Steele
936b8a289c Allow separate paragraphs in release items.
The first paragraph should match the first line of the commit message as closely as possible.  The following paragraphs add more information.

Release items have been updated back to 2.01.
2019-05-21 10:37:30 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
David Steele
1bc84c6474 The local command for restore is implemented entirely in C.
This is just the part of restore run by the local helper processes, not the entire command.

Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 17:07:37 -04:00
Cynthia Shang
a839830333 Add most unimplemented functions to the remote storage driver.
Add pathCreate(), pathRemove(), pathSync(), and remove() to the driver.

Contributed by Cynthia Shang.
2019-05-20 16:19:14 -04:00
David Steele
a9f589b87a Begin v2.15 development. 2019-05-20 12:23:40 -04:00
David Steele
0f685a33a0 v2.14: Bug Fix and Improvements
Bug Fixes:

* Fix segfault when process-max > 8 for archive-push/archive-get. (Reported by Jens Wilke.)

Improvements:

* Bypass database checks when stanza-delete issued with force. (Contributed by Cynthia Shang. Suggested by hatifnatt.)
* Add configure script for improved multi-platform support.

Documentation Features:

* Add user guides for CentOS/RHEL 6/7.
2019-05-20 11:51:58 -04:00
David Steele
51d67ce1ba Update contributor name. 2019-05-16 09:59:34 -04:00
Cynthia Shang
19d8358cba Update mock/expire module test matrix so expect tests output.
Also add an error message to prevent regression.

Contributed by Cynthia Shang.
2019-05-16 09:53:55 -04:00
David Steele
c51274d1b6 Add user guides for CentOS/RHEL 6/7.
It would be better if the documentation could be generated on multiple operating systems all in one go, but the doc system currently does not allow vars to be changed once they are set.

The solution is to run the docs for each required OS and stitch the documentation together.  It's not pretty but it works and the automation in release.pl should at least make it easy to use.
2019-05-16 08:32:02 -04:00
David Steele
bc7b42e718 Restore index menu url default lost in b85e51d6.
The url for the menu item referring to the index (i.e. site root page) should use {[project-url-root]}.

This allows the url to be set to different values depending on the location of the index.
2019-05-16 07:35:45 -04:00
David Steele
a4561dc087 Add --out-preserve to preserve contents of output path.
Useful for building a single set of documentation over multiple doc.pl runs.
2019-05-16 07:29:17 -04:00
Cynthia Shang
18d4cb5741 Bypass database checks when stanza-delete issued with force.
Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted.

Contributed by Cynthia Shang.
Suggested by Roman.
2019-05-15 13:14:58 -04:00
David Steele
5c1d4bcd0d Automate coverage summary report generation.
This report replaces the lcov report that was generated manually for each release.

The lcov report was overly verbose just to say that we have virtually 100% coverage.
2019-05-15 13:04:56 -04:00
David Steele
5bba72b874 Remove -Wswitch-enum compiler option.
The -Wswitch option included in -Wall provides the same level of coverage and allows enum options to be grouped into default.
2019-05-15 12:55:08 -04:00
David Steele
53f6232264 Improve coverage in config/parse module. 2019-05-14 10:35:39 -04:00
David Steele
1e5ab69cad Improve coverage in config/config module. 2019-05-13 19:10:24 -04:00
David Steele
15a33bf74b Error on multiple option alternate names and simplify help command.
There are currently no options with multiple alternate (deprecated) names so the code to render them in the help command could not be covered.

Remove the uncovered code and add an error when multiple alternate names are configured.  It's not clear that the current code was handling this correctly, so it will need to be reviewed if it comes up again.
2019-05-13 17:10:41 -04:00
David Steele
2d2bec842a Improve coverage in perl/exec module. 2019-05-13 13:36:24 -04:00
David Steele
431532574b Add testUser() and testGroup().
Retrieve the name of the current user/group used for testing.
2019-05-13 07:51:11 -04:00
David Steele
31d0fe5f50 Improve log performance, simplify macros, rename logWill() to logAny().
Pre-calculate the value used by logAny() to improve performance and make it more likely to be inlined.

Move IF_LOG_ANY() into LOG_INTERNAL() to simplify the macros and improve performance of LOG() and LOG_PID().  If the message has no chance of being logged there's no reason to call logInternal().

Rename logWill() to logAny() because it seems more intuitive.
2019-05-11 18:20:57 -04:00