1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3312 Commits

Author SHA1 Message Date
David Steele
d859fe8c4d Update to remove files after archive-get test completes.
This cleans up the spool path for the next test. Currently there isn't one but there will be.
2021-04-14 15:48:04 -04:00
David Steele
040ad71f8c Remove lockClear().
This function has not been used since the switch to the fork/exec model.

lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
David Steele
c2d4a0286e Define DEBUG in build.auto.c.
Both NDEBUG and DEBUG were used in the code, which was a bit confusing.

Define DEBUG in build.auto.c so it is available in all C and header files and stop using NDEBUG. This is preferable to using NDEBUG everywhere since there are multiple DEBUG* defines, e.g. DEBUG_COVERAGE.

Note that NDEBUG is still required since it is used by the C libraries.
2021-04-13 18:06:07 -04:00
David Steele
c1aae434ca Update Cirrus CI FreeBSD release to 12.2. 2021-04-13 17:03:28 -04:00
David Steele
a0e24d492f Add THIS_PUB() macro to simplify inline object accessors.
In combination with the thisPub() function, this macro simplifies accessing the public part of a private object struct.

thisPub() asserts this != NULL so the caller does not need to do it.
2021-04-13 16:58:02 -04:00
David Steele
9fec4ce98c Refactor remaining common/io modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the remaining common/io modules.
2021-04-13 14:37:02 -04:00
Isaacwhyuenac
5bf160643b Clarify that repo-s3-role is not an ARN. 2021-04-13 14:02:20 -04:00
David Steele
2c9c59170f Add bash to Cirrus CI FreeBSD package install.
It appears bash was removed from the base image so install explicitly.
2021-04-13 12:38:17 -04:00
David Steele
8844ced384 Refactor common/io/filter module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/io/filter module.
2021-04-12 16:05:40 -04:00
David Steele
3b9bed9518 Refactor common/type/keyValue module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/type/keyValue module.
2021-04-09 17:29:14 -04:00
David Steele
c434b9ed88 Refactor common/type/xml module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/type/xml module.
2021-04-09 17:03:55 -04:00
David Steele
e18f3c1701 Remove unused XML constructors.
These alternate constructors may have been used in the Perl days but they are no longer used so remove them.
2021-04-09 15:56:42 -04:00
David Steele
ddd37ebf7d Refactor common/exec module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/exec module.
2021-04-09 15:31:31 -04:00
David Steele
1d674c352f Refactor db module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the db module.
2021-04-09 14:32:21 -04:00
David Steele
442b2e41b1 Refactor info modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the info modules.
2021-04-09 13:48:40 -04:00
David Steele
4937653a3d Refactor protocol modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
David Steele
235e32b57d Add const keyword to module scope constants.
These pointers should never be modified, so mark them const.
2021-04-08 14:47:53 -04:00
David Steele
6c005afd5b Add Ubuntu 20.04 test VM definition. 2021-04-08 14:38:20 -04:00
David Steele
d30ec9c9ae Replace OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() with inlines.
Inline functions are more efficient and if they are not used are automatically omitted from the binary.

This also makes the implementation of these functions easier to find and removes the need for a declaration. That is, the complete implementation is located in the header rather than being spread between the header and C file.
2021-04-08 10:04:57 -04:00
David Steele
351e7db4c4 Replace OBJECT_DEFINE_FREE_RESOURCE_BEGIN() with normal functions.
OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() will be replaced with inlines so this would be the only macro left that is constructing functions.

It is not a great pattern anyway since it makes it hard to find the function implementation.
2021-04-07 16:27:55 -04:00
David Steele
cc85c4f03d Replace OBJECT_DEFINE_GET() with *Pub struct pattern.
This macro was originally intended to simplify the creation of simple getters but it has been superseded by the pattern introduced in 79a2d02c.

Remove instances of OBJECT_DEFINE_GET() to avoid confusion with the new pattern.
2021-04-07 14:27:57 -04:00
David Steele
b715c70b46 Refactor storage modules with inline getters/setters.
Extended the pattern introduced in 79a2d02c to the storage modules: Storage, StorageRead, StorageWrite.
2021-04-07 14:04:38 -04:00
David Steele
79a2d02c9c Refactor List, StringList, and VariantList for performance.
Introduce a standard pattern for exposing public struct members (as documented in CODING.md) and use it to inline lstSize() which should improve the performance of iterating large lists.

Since many functions in these modules are just thin wrappers of other functions, inline where appropriate.

Remove strLstExistsZ() and strLstInsertZ() since they were only used in tests, where the String version of the function is sufficient.

Move strLstNewSplitSizeZ() to command/help/help.c and remove strLstNewSplitSize(). This function has only ever been used by help and does not seem widely applicable.
2021-04-07 12:50:33 -04:00
David Steele
904738a5f1 Update config.sub to latest version. 2021-04-05 10:25:01 -04:00
David Steele
8461961e5c Begin v2.34 development. 2021-04-05 10:23:16 -04:00
David Steele
a3d89143d6 v2.33: Multi-Repository and GCS Support
Bug Fixes:

* Fix option warnings breaking async archive-get/archive-push. (Reviewed by Cynthia Shang. Reported by Lev Kokotov.)
* Fix memory leak in backup during archive copy. (Reviewed by Cynthia Shang. Reported by Christian ROUX, Efremov Egor.)
* Fix stack overflow in cipher passphrase generation. (Reviewed by Cynthia Shang. Reported by bsiara.)
* Fix repo-ls / on S3 repositories. (Reviewed by Cynthia Shang. Reported by Lesovsky Alexey.)

Features:

* Multiple repository support. (Contributed by Cynthia Shang, David Steele. Reviewed by Stefan Fercot, Stephen Frost.)
* GCS support for repository storage. (Reviewed by Cynthia Shang.)
* Add archive-header-check option. (Reviewed by Stephen Frost, Cynthia Shang. Suggested by Hans-Jürgen Schönig.)

Improvements:

* Include recreated system databases during selective restore. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Exclude content-length from S3 signed headers. (Reviewed by Cynthia Shang. Suggested by Brian P Bockelman.)
* Consolidate less commonly used repository storage options. (Reviewed by Cynthia Shang.)
* Allow custom config-path default with ./configure --with-configdir. (Contributed by Michael Schout. Reviewed by David Steele.)
* Log archive copy during backup. (Reviewed by Cynthia Shang, Stefan Fercot.)

Documentation Improvements:

* Update reference to include links to user guide examples. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Update selective restore documentation with caveats. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add compress-type clarification to archive-copy documentation. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add compress-level defaults per compress-type value. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Add note about required NFS settings being the same as PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-04-05 09:18:20 -04:00
David Steele
a6a544c7f9 Fix extraneous hrnReplaceKey() calls in unit tests.
hrnReplaceKey() was added to the TEST_ERROR*() macros in 58760486 but some calls to TEST_ERROR*() already used it. This led to the function being called twice on the same buffer which had no effect but valgrind definitely did not like.

Remove extraneous calls to make valgrind happy. Since this is test code there are no implications for production.
2021-04-02 15:35:41 -04:00
David Steele
1615cb0261 Add --gen-check option to test.pl to check autogenerated code.
Make sure that auto-generated code does not change during CI.

This is useful for catching missed code generation, especially help.auto.c.
2021-03-31 10:36:22 -04:00
David Steele
e89452557e Update to help.auto.c missed in d372dd652c. 2021-03-31 09:41:33 -04:00
Cynthia Shang
d372dd652c
Update reference to include links to user guide examples.
The command-example and command-example-list elements were removed from the documentation rendering some time ago so these tags were dead code. The tags, however, contained some examples and information that were pertinent to the command, so where possible, the information was included in the description of the command and/or the user-guide and links to the relevant user guide sections were added.

Note that some commands could not be updated with user guide references since doing so would cause a cyclical reference in the user guide. These commands have an internal comment to indicate this.

In addition, some clarifications were added (e.g. expire --set option) where information was lacking.
2021-03-31 09:36:56 -04:00
David Steele
fead2360da Link Github issues/PRs to release notes.
This makes it easier to determine which release notes relate to issues and PRs, especially for bug fixes or user requests.
2021-03-30 15:49:03 -04:00
David Steele
21db7f65ef Update to help.auto.c missed in 75987621. 2021-03-29 09:49:42 -04:00
Cynthia Shang
75987621fa
Add note about required NFS settings being the same as PostgreSQL. 2021-03-26 10:11:06 -04:00
Cynthia Shang
3e206088e7
Add compress-level defaults per compress-type value.
Document these defaults until they can be added to the config parser and automated.
2021-03-26 09:25:31 -04:00
David Steele
b6106f3c1f
Add archive-header-check option.
Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.

Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
David Steele
01b8e2258f
Improve archive-push command fault tolerance.
3b8f0ef missed some cases that could cause archive-push to fail:

* Checking archive info.
* Checking to see if a WAL segment already exists.

These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
Cynthia Shang
2789d3b620
Improve info command fault tolerance.
This improvement reduces the number of errors thrown; these errors will now be reported as a status for the stanza or repo as appropriate. Invalid option configurations are still thrown but all other errors are caught, formatted and reported. This was necessary for multiple repositories so that the command can complete gathering information from each repository and report the results rather than immediately aborting when an error occurs.

Two new error codes were introduced:
6 = requested backup not found
99 = other, which is used to indicate an error has occurred that requires more details to be provided

A new stanza name of "[invalid]" was created for instances where a stanza was not specified and no stanza can be found.

If there is only one repository configured the error will move up to the stanza level with the standard error formatting of 'error (message)' where the message will be "other" and the details of the error will be listed on the next line(s):

stanza: stanza1
    status: error (other)
            [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
            CryptoError: cipher header invalid
            HINT: is or was the repo encrypted?
            FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
            HINT: backup.info cannot be opened and is required to perform a backup.
            HINT: has a stanza-create been performed?
            HINT: use option --stanza if encryption settings are different for the stanza than the global
    cipher: aes-256-cbc

If a backup set is requested but is not found on any repo, a stanza-level status error of 'requested backup not found' is reported when there are no other errors:

pgbackrest info --stanza=demo --set=bogus
stanza: demo
    status: error (requested backup not found)
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

If there are multiple repositories configured and a single repo is in error but the other repos are ok or have a different error:

pgbackrest info --stanza=demo --set=20210322-171211F
stanza: demo
    status: mixed
        repo1: error
               [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
               CryptoError: cipher header invalid
               HINT: is or was the repo encrypted?
               FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
               HINT: backup.info cannot be opened and is required to perform a backup.
               HINT: has a stanza-create been performed?
               HINT: use option --stanza if encryption settings are different for the stanza than the global
        repo2: ok
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

    db (current)
        wal archive min/max (12): 000000010000000000000001/000000010000000000000003

        full backup: 20210322-171211F
            timestamp start/stop: 2021-03-22 17:12:11 / 2021-03-22 17:12:28
            wal start/stop: 000000010000000000000002 / 000000010000000000000002
            database size: 23.4MB, database backup size: 23.4MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB
            database list: postgres (13359)

Json output will include the repository information and any error information. If no stanzas are found, then [invalid] will be set as the name:

[
   {
       "archive":[],
       "backup":[],
       "cipher":"none",
       "db":[],
       "name":"[invalid]",
       "repo":[
           {
               "cipher":"none",
               "key":1,
               "status":{
                   "code":99,
                   "message":"[PathOpenError] unable to list file info for path '/var/lib/pgbackrest/repo2/backup': [13] Permission denied"
               }
           }
       ],
       "status":{
           "code":99,
           "lock":{"backup":{"held":false}},
           "message":"other"
           }
   }
]
2021-03-25 12:29:36 -04:00
David Steele
7d7ac0e0eb
Exclude content-length from S3 signed headers.
The content-length header was being signed since it was the only header that didn't need to be and it seemed simpler just to sign it as well. Also, the S3 documentation encourages signing as many headers as possible to avoid tampering.

However, some proxies munge this header causing authentication failure, so skip signing content-length.
2021-03-25 07:07:16 -04:00
David Steele
5876048675 Allow key replacements in TEST_ERROR*() macros.
Allow standard replacements (e.g. {[path]}) in TEST_ERROR*() macros.

Update command/archive-push unit test as an example of usage.
2021-03-23 18:20:26 -04:00
David Steele
b2440947fb Fix comment type and phrasing. 2021-03-18 16:29:14 -04:00
David Steele
6800f9e9cd Changed unnecessary strNewFmt() to strNew(). 2021-03-18 16:26:30 -04:00
David Steele
73e94a4e9c Fix comment typo. 2021-03-16 17:02:03 -04:00
David Steele
2016fac0d9
Improve protocol handlers.
Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
David Steele
b1d945ebb7
Fix repo-ls / on S3 repositories.
S3 returns 200 for HEAD / which indicates it is a file but does not return the expected headers which causes an error.

Rather than fix this for S3, just automatically return / as not existing for any storage that does not support paths.

Also add some defensive checks to prevent this from generating a segfault if it happens again.
2021-03-16 12:42:55 -04:00
David Steele
0eb5b22ffa Add back performance tests lost during Github Actions switch.
These tests do not provide code coverage but testing them regularly is important so they work when needed.
2021-03-15 17:24:58 -04:00
Stefan Fercot
6942ff569d
Include recreated system databases during selective restore.
Some standard system databases (e.g. postgres) may be recreated by the user and have an OID that makes them look like user databases.

Identify the standard three system databases (template0, template1, postgres) and restore them non-zeroed no matter what OID they have.
2021-03-15 12:54:14 -04:00
David Steele
c0283eec99 Explicitly free local processes after restore error unit test.
Local processes are still running after this error and it is best to free them before ending the test.
2021-03-12 17:13:34 -05:00
David Steele
ec347847e5 Pass cipher type directly to backupFileProtocol().
Cipher type was inferred from the presence of cipherSubPass rather than being passed explicitly in order to maintain compatibility with Perl backupFile().

Now that Perl is gone it makes sense to pass it explicitly, as we do elsewhere.
2021-03-12 15:19:32 -05:00
David Steele
e07040c2e4 Add HRN_STORAGE_TIME() harness macro.
Makes updating the time of a path/file more streamlined in tests.

Also update all tests where utime() was being used directly.
2021-03-12 12:54:34 -05:00
David Steele
3c85a497a6 Add backup delta unit test.
This test was added to take the place of another test, which turned out not to be workable.

Even so, it adds coverages at little cost so it seems worth keeping.
2021-03-11 14:40:14 -05:00