1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2237 Commits

Author SHA1 Message Date
David Steele
44eb21ea93 Use HEAD to check if a file exists on S3.
The previous implementation searched for the file in a list which worked but was not optimal.  For arbitrary bucket structures it would also produce a false negative if a match was not found in the first 1000 entries.  This was not an issue for our repo structure since the max hits on exists calls is two but it seems worth fixing to avoid future complications.
2019-06-04 10:05:27 -04:00
David Steele
15b8e3b6af Make C S3 requests use the same host logic as Perl.
The C code was passing the host (if specified) with the request which could force the server into path-style URLs, which are not supported.

Instead, use the Perl logic of always passing bucket.endpoint in the request no matter what host is used for the HTTPS connection.

It's an open question whether we should support path-style URLs but since we don't it's useless to tell the server otherwise.  Note that Amazon S3 has deprecated path-style URLs and they are no longer supported on newly created buckets.
2019-06-04 09:39:08 -04:00
David Steele
5f92c36b30 Fix erroneous parameter name.
The exists() function only checks for files so the parameter name should reflect that.
2019-06-04 08:04:33 -04:00
David Steele
0945d9666b Bring back PATH_PAX for platforms that don't define it.
This define was replaced in 8c712d89 with limits.h but that caused an issue with the hurd-i386 build for Debian which apparently does not define this value.
2019-06-01 11:09:01 -04:00
David Steele
a207af7921 Fix mismatched log/test macros. 2019-06-01 09:44:40 -04:00
David Steele
12bca3c43e Add CPPFLAGS to compile rules.
This should silence the last of the Debian package warnings.
2019-06-01 09:28:31 -04:00
David Steele
388ba0458c Fix build.flags being removed on each build.
This was being removed by rsync which forced a full build even when a partial should have been fine.  Rewrite the file after the rsync so it is preserved.
2019-05-31 18:37:31 -04:00
David Steele
6cba50c3f2 Remove Debian package patch merged to upstream. 2019-05-31 18:32:40 -04:00
David Steele
3a5fd7b28a Fix typo in macro name. 2019-05-30 11:19:59 -04:00
David Steele
a2ec1253e9 Add code classification exclusion missed in 3e1b06ac. 2019-05-30 10:44:35 -04:00
David Steele
64260b2e98 Build all docs with S3 using --var=s3-all=y
Force repo-type=s3 for all tests.  This is not currently the default for any OS builds.
2019-05-29 08:38:45 -04:00
David Steele
404284b90f Add internal flag for commands.
Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.

Update help for ls now that it is exposed.
2019-05-28 12:18:05 -04:00
David Steele
20e5b92f36 Add ls command.
Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.

This command is internal-only so the interface may change at any time without notice.
2019-05-28 10:03:48 -04:00
David Steele
3b3327eae6 Move tls/http statistics output to command/command.
This module already has the filtering required to keep these messages from being displayed by default for commands that output to stdout (e.g. info).
2019-05-28 09:50:59 -04:00
David Steele
3e1b06acaa Use minio as local S3 emulator in documentation.
The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.

Also, install certificates and CA so that TLS verification can be enabled.
2019-05-27 07:37:20 -04:00
David Steele
a474ba54c5 Refactoring path support in the storage module.
Not all storage types support paths as a physical thing that must be created/destroyed.  Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object.

Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense.

Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.
2019-05-26 12:41:15 -04:00
David Steele
38f28bd520 Log TLS and HTTP statistics on exit.
These stats measure how efficiently TLS and HTTP are reusing connections (i.e. pipelining).
2019-05-26 12:32:49 -04:00
David Steele
819eda0818 Set log level of protocol processing functions to debug.
Setting these to trace effectively made debug level useless in local/remote processes since all debug messages were demoted to trace when called from these functions.
2019-05-26 12:29:29 -04:00
David Steele
d12d94c53c Make info(), pathCreate() and pathSync() optional for storage drivers.
These functions are not required for repository storage so make them optional and error if they are not implemented for non-repository storage, .e.g. pg or spool.

The goal is to simplify the drivers (e.g. S3) that are intended only for repository storage.
2019-05-24 18:09:16 -04:00
David Steele
39645fc1a9 Add pathRemove() and remove() to S3 storage.
These functions will be needed for the expire command.
2019-05-24 14:33:47 -04:00
David Steele
6c385dfa7a Add function to determine if HTTP response code is OK.
OK responses are a range rather than a single value, so simplify the check with a function.
2019-05-24 13:56:38 -04:00
David Steele
96770c529b storageList() returns an empty list by default for missing paths.
The prior behavior was to return NULL so the caller would know the path was missing, but this is rarely useful, complicates the calling code, and increases the chance of segfaults.

The .nullOnMissing param has been added to enable the prior behavior.
2019-05-24 13:12:56 -04:00
David Steele
707e4a29f0 Notify compiler that cryptoErrorCode() does not return. 2019-05-24 11:09:43 -04:00
David Steele
01f3024b5c Add missing const qualifier. 2019-05-24 11:06:11 -04:00
David Steele
39cb624831 Add missing menus to the new user guides.
Since the CentOS 6/7 user guides were generated as a single page they did not get menus.  Generate the entire site for each user guide so menus are included.
2019-05-24 07:45:03 -04:00
David Steele
04f8b4ea52 Use git url for website clone. 2019-05-24 07:41:55 -04:00
David Steele
ec9622cde8 Use the git log to ease release note management.
The release notes are generally a direct reflection of the git log.  So, ease the burden of maintaining the release notes by using the git log to determine what needs to be added.

Currently only non-dev items are required to be matched to a git commit but the goal is to account for all commits.

The git history cache is generated from the git log but can be modified to correct typos and match the release notes as they evolve.  The commit hash is used to identify commits that have already been added to the cache.

There's plenty more to do here.  For instance, links to the commits for each release item should be added to the release notes.
2019-05-22 18:54:49 -04:00
David Steele
86482c7db9 Reduce log level for all expect tests to detail.
The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.

Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-22 18:23:44 -04:00
David Steele
e4cc008b98 Remove duplicated hint.
At some point this hint got added to the underlying code so it is no longer needed here.
2019-05-22 11:31:38 -04:00
David Steele
ff9c1bed5d Save cipher-pass key/value missed in f492f057.
This value is required when encryption is enabled.

In passing simplify the expression used to skip the checksum when calculating the checksum.
2019-05-22 11:24:18 -04:00
David Steele
3da60f4b5e Don't escape / when rendering JSON.
/ is escaped in the spec but the Perl renderer we use does not escape it which leads to checksum mismatches between the two sets of code.

This particular escape seems to be a more recent addition to the spec and is targeted toward embedding JSON in JavaScript.

\/ is still allowed when parsing JSON.
2019-05-22 11:10:43 -04:00
David Steele
664054fea9 Remove RHEL package patch merged to upstream. 2019-05-22 06:55:31 -04:00
David Steele
e1cce9c614 Remove extraneous parens. 2019-05-21 14:07:35 -04:00
David Steele
936b8a289c Allow separate paragraphs in release items.
The first paragraph should match the first line of the commit message as closely as possible.  The following paragraphs add more information.

Release items have been updated back to 2.01.
2019-05-21 10:37:30 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
David Steele
1bc84c6474 The local command for restore is implemented entirely in C.
This is just the part of restore run by the local helper processes, not the entire command.

Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 17:07:37 -04:00
Cynthia Shang
a839830333 Add most unimplemented functions to the remote storage driver.
Add pathCreate(), pathRemove(), pathSync(), and remove() to the driver.

Contributed by Cynthia Shang.
2019-05-20 16:19:14 -04:00
David Steele
bbf2e0d5b0 Remove extra linefeed. 2019-05-20 12:24:43 -04:00
David Steele
a9f589b87a Begin v2.15 development. 2019-05-20 12:23:40 -04:00
David Steele
0f685a33a0 v2.14: Bug Fix and Improvements
Bug Fixes:

* Fix segfault when process-max > 8 for archive-push/archive-get. (Reported by Jens Wilke.)

Improvements:

* Bypass database checks when stanza-delete issued with force. (Contributed by Cynthia Shang. Suggested by hatifnatt.)
* Add configure script for improved multi-platform support.

Documentation Features:

* Add user guides for CentOS/RHEL 6/7.
2019-05-20 11:51:58 -04:00
David Steele
51d67ce1ba Update contributor name. 2019-05-16 09:59:34 -04:00
Cynthia Shang
2b75f57232 Add linefeeds for clarity. 2019-05-16 09:56:48 -04:00
Cynthia Shang
19d8358cba Update mock/expire module test matrix so expect tests output.
Also add an error message to prevent regression.

Contributed by Cynthia Shang.
2019-05-16 09:53:55 -04:00
David Steele
c51274d1b6 Add user guides for CentOS/RHEL 6/7.
It would be better if the documentation could be generated on multiple operating systems all in one go, but the doc system currently does not allow vars to be changed once they are set.

The solution is to run the docs for each required OS and stitch the documentation together.  It's not pretty but it works and the automation in release.pl should at least make it easy to use.
2019-05-16 08:32:02 -04:00
David Steele
bc7b42e718 Restore index menu url default lost in b85e51d6.
The url for the menu item referring to the index (i.e. site root page) should use {[project-url-root]}.

This allows the url to be set to different values depending on the location of the index.
2019-05-16 07:35:45 -04:00
David Steele
a4561dc087 Add --out-preserve to preserve contents of output path.
Useful for building a single set of documentation over multiple doc.pl runs.
2019-05-16 07:29:17 -04:00
David Steele
bc30027d73 Fix typo in doc.pl command-line help. 2019-05-16 07:04:28 -04:00
Cynthia Shang
18d4cb5741 Bypass database checks when stanza-delete issued with force.
Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted.

Contributed by Cynthia Shang.
Suggested by Roman.
2019-05-15 13:14:58 -04:00
David Steele
5c1d4bcd0d Automate coverage summary report generation.
This report replaces the lcov report that was generated manually for each release.

The lcov report was overly verbose just to say that we have virtually 100% coverage.
2019-05-15 13:04:56 -04:00
David Steele
5bba72b874 Remove -Wswitch-enum compiler option.
The -Wswitch option included in -Wall provides the same level of coverage and allows enum options to be grouped into default.
2019-05-15 12:55:08 -04:00