1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-03 00:26:59 +02:00
Commit Graph

2400 Commits

Author SHA1 Message Date
6e809e578f Add tag to specify minio version to use for documentation build.
The new minio major release broke the build.  We'll need to figure that out but for now use the last major version, which is known to work.
2019-06-11 10:34:42 -04:00
7f2f535460 Add info() and infoList() to S3 driver.
These should be the last functions required to complete the implementation of the S3 driver.
2019-06-10 16:09:38 -04:00
9d1b03781f Revert removal of aws cli in 3e1b06ac.
This is required for integration tests to create buckets.
2019-06-10 15:52:51 -04:00
456391f67c Create general-purpose S3 list function.
There are already two functions which need file lists and more on the way, so create a general-purpose function to reduce code duplication.
2019-06-10 12:23:04 -04:00
fb8bd095ca Add missing const qualifiers. 2019-06-10 10:37:20 -04:00
78b333790c Return IoRead/IoWrite from io*FilterGroupSet()
This makes is possible to chain functions more effectively.
2019-06-07 10:35:44 -04:00
e701e8fd84 Expose storage type.
Code calling the storage should be driver agnostic but knowing the storage type can be useful for debugging and error reporting.
2019-06-05 11:50:49 -04:00
6ff3325c77 Enforce requiring repo-cipher-pass at config parse time.
This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.

Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
2019-06-05 11:43:17 -04:00
d7bd0c58cd Use wal_level=replica in the documentation for PostgreSQL >= 9.6.
The documentation was using wal_level=hot_standby which is a deprecated setting.

Also remove the reference to wal_level=archive since it is no longer supported and is not recommended for older versions.

Suggested by Patrick McLaughlin.
2019-06-05 07:27:24 -04:00
aca11b2fa1 Add String to types supported by JSON list render.
All types should be supported eventually but for now this answers the need.
2019-06-04 17:40:27 -04:00
0ab6f3bb87 Fix incorrect error type on missing path. 2019-06-04 13:38:05 -04:00
4b91259de8 Make working with filter groups less restrictive.
Filter groups could not be manipulated once they had been assigned to an IO object.  Now they can be freely manipulated up to the time the IO object is opened.

Also, move the filter group into the IO object's context so they don't need to be tracked separately.
2019-06-04 12:56:04 -04:00
92e04ea9f4 Remove per-stanza repo cache clear during testing.
This was not being used and is not supported by the equivalent C code.
2019-06-04 10:34:19 -04:00
44eb21ea93 Use HEAD to check if a file exists on S3.
The previous implementation searched for the file in a list which worked but was not optimal.  For arbitrary bucket structures it would also produce a false negative if a match was not found in the first 1000 entries.  This was not an issue for our repo structure since the max hits on exists calls is two but it seems worth fixing to avoid future complications.
2019-06-04 10:05:27 -04:00
15b8e3b6af Make C S3 requests use the same host logic as Perl.
The C code was passing the host (if specified) with the request which could force the server into path-style URLs, which are not supported.

Instead, use the Perl logic of always passing bucket.endpoint in the request no matter what host is used for the HTTPS connection.

It's an open question whether we should support path-style URLs but since we don't it's useless to tell the server otherwise.  Note that Amazon S3 has deprecated path-style URLs and they are no longer supported on newly created buckets.
2019-06-04 09:39:08 -04:00
5f92c36b30 Fix erroneous parameter name.
The exists() function only checks for files so the parameter name should reflect that.
2019-06-04 08:04:33 -04:00
0945d9666b Bring back PATH_PAX for platforms that don't define it.
This define was replaced in 8c712d89 with limits.h but that caused an issue with the hurd-i386 build for Debian which apparently does not define this value.
2019-06-01 11:09:01 -04:00
a207af7921 Fix mismatched log/test macros. 2019-06-01 09:44:40 -04:00
12bca3c43e Add CPPFLAGS to compile rules.
This should silence the last of the Debian package warnings.
2019-06-01 09:28:31 -04:00
388ba0458c Fix build.flags being removed on each build.
This was being removed by rsync which forced a full build even when a partial should have been fine.  Rewrite the file after the rsync so it is preserved.
2019-05-31 18:37:31 -04:00
6cba50c3f2 Remove Debian package patch merged to upstream. 2019-05-31 18:32:40 -04:00
3a5fd7b28a Fix typo in macro name. 2019-05-30 11:19:59 -04:00
a2ec1253e9 Add code classification exclusion missed in 3e1b06ac. 2019-05-30 10:44:35 -04:00
64260b2e98 Build all docs with S3 using --var=s3-all=y
Force repo-type=s3 for all tests.  This is not currently the default for any OS builds.
2019-05-29 08:38:45 -04:00
404284b90f Add internal flag for commands.
Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.

Update help for ls now that it is exposed.
2019-05-28 12:18:05 -04:00
20e5b92f36 Add ls command.
Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.

This command is internal-only so the interface may change at any time without notice.
2019-05-28 10:03:48 -04:00
3b3327eae6 Move tls/http statistics output to command/command.
This module already has the filtering required to keep these messages from being displayed by default for commands that output to stdout (e.g. info).
2019-05-28 09:50:59 -04:00
3e1b06acaa Use minio as local S3 emulator in documentation.
The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.

Also, install certificates and CA so that TLS verification can be enabled.
2019-05-27 07:37:20 -04:00
a474ba54c5 Refactoring path support in the storage module.
Not all storage types support paths as a physical thing that must be created/destroyed.  Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object.

Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense.

Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.
2019-05-26 12:41:15 -04:00
38f28bd520 Log TLS and HTTP statistics on exit.
These stats measure how efficiently TLS and HTTP are reusing connections (i.e. pipelining).
2019-05-26 12:32:49 -04:00
819eda0818 Set log level of protocol processing functions to debug.
Setting these to trace effectively made debug level useless in local/remote processes since all debug messages were demoted to trace when called from these functions.
2019-05-26 12:29:29 -04:00
d12d94c53c Make info(), pathCreate() and pathSync() optional for storage drivers.
These functions are not required for repository storage so make them optional and error if they are not implemented for non-repository storage, .e.g. pg or spool.

The goal is to simplify the drivers (e.g. S3) that are intended only for repository storage.
2019-05-24 18:09:16 -04:00
39645fc1a9 Add pathRemove() and remove() to S3 storage.
These functions will be needed for the expire command.
2019-05-24 14:33:47 -04:00
6c385dfa7a Add function to determine if HTTP response code is OK.
OK responses are a range rather than a single value, so simplify the check with a function.
2019-05-24 13:56:38 -04:00
96770c529b storageList() returns an empty list by default for missing paths.
The prior behavior was to return NULL so the caller would know the path was missing, but this is rarely useful, complicates the calling code, and increases the chance of segfaults.

The .nullOnMissing param has been added to enable the prior behavior.
2019-05-24 13:12:56 -04:00
707e4a29f0 Notify compiler that cryptoErrorCode() does not return. 2019-05-24 11:09:43 -04:00
01f3024b5c Add missing const qualifier. 2019-05-24 11:06:11 -04:00
39cb624831 Add missing menus to the new user guides.
Since the CentOS 6/7 user guides were generated as a single page they did not get menus.  Generate the entire site for each user guide so menus are included.
2019-05-24 07:45:03 -04:00
04f8b4ea52 Use git url for website clone. 2019-05-24 07:41:55 -04:00
ec9622cde8 Use the git log to ease release note management.
The release notes are generally a direct reflection of the git log.  So, ease the burden of maintaining the release notes by using the git log to determine what needs to be added.

Currently only non-dev items are required to be matched to a git commit but the goal is to account for all commits.

The git history cache is generated from the git log but can be modified to correct typos and match the release notes as they evolve.  The commit hash is used to identify commits that have already been added to the cache.

There's plenty more to do here.  For instance, links to the commits for each release item should be added to the release notes.
2019-05-22 18:54:49 -04:00
86482c7db9 Reduce log level for all expect tests to detail.
The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.

Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-22 18:23:44 -04:00
e4cc008b98 Remove duplicated hint.
At some point this hint got added to the underlying code so it is no longer needed here.
2019-05-22 11:31:38 -04:00
ff9c1bed5d Save cipher-pass key/value missed in f492f057.
This value is required when encryption is enabled.

In passing simplify the expression used to skip the checksum when calculating the checksum.
2019-05-22 11:24:18 -04:00
3da60f4b5e Don't escape / when rendering JSON.
/ is escaped in the spec but the Perl renderer we use does not escape it which leads to checksum mismatches between the two sets of code.

This particular escape seems to be a more recent addition to the spec and is targeted toward embedding JSON in JavaScript.

\/ is still allowed when parsing JSON.
2019-05-22 11:10:43 -04:00
664054fea9 Remove RHEL package patch merged to upstream. 2019-05-22 06:55:31 -04:00
e1cce9c614 Remove extraneous parens. 2019-05-21 14:07:35 -04:00
936b8a289c Allow separate paragraphs in release items.
The first paragraph should match the first line of the commit message as closely as possible.  The following paragraphs add more information.

Release items have been updated back to 2.01.
2019-05-21 10:37:30 -04:00
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
1bc84c6474 The local command for restore is implemented entirely in C.
This is just the part of restore run by the local helper processes, not the entire command.

Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 17:07:37 -04:00
a839830333 Add most unimplemented functions to the remote storage driver.
Add pathCreate(), pathRemove(), pathSync(), and remove() to the driver.

Contributed by Cynthia Shang.
2019-05-20 16:19:14 -04:00