db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-15 01:04:37 +02:00
Code Issues Releases Activity

Improve info command fault tolerance.

Browse Source 
This improvement reduces the number of errors thrown; these errors will now be reported as a status for the stanza or repo as appropriate. Invalid option configurations are still thrown but all other errors are caught, formatted and reported. This was necessary for multiple repositories so that the command can complete gathering information from each repository and report the results rather than immediately aborting when an error occurs.

Two new error codes were introduced:
6 = requested backup not found
99 = other, which is used to indicate an error has occurred that requires more details to be provided

A new stanza name of "[invalid]" was created for instances where a stanza was not specified and no stanza can be found.

If there is only one repository configured the error will move up to the stanza level with the standard error formatting of 'error (message)' where the message will be "other" and the details of the error will be listed on the next line(s):

stanza: stanza1
    status: error (other)
            [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
            CryptoError: cipher header invalid
            HINT: is or was the repo encrypted?
            FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
            HINT: backup.info cannot be opened and is required to perform a backup.
            HINT: has a stanza-create been performed?
            HINT: use option --stanza if encryption settings are different for the stanza than the global
    cipher: aes-256-cbc

If a backup set is requested but is not found on any repo, a stanza-level status error of 'requested backup not found' is reported when there are no other errors:

pgbackrest info --stanza=demo --set=bogus
stanza: demo
    status: error (requested backup not found)
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

If there are multiple repositories configured and a single repo is in error but the other repos are ok or have a different error:

pgbackrest info --stanza=demo --set=20210322-171211F
stanza: demo
    status: mixed
        repo1: error
               [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
               CryptoError: cipher header invalid
               HINT: is or was the repo encrypted?
               FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
               HINT: backup.info cannot be opened and is required to perform a backup.
               HINT: has a stanza-create been performed?
               HINT: use option --stanza if encryption settings are different for the stanza than the global
        repo2: ok
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

    db (current)
        wal archive min/max (12): 000000010000000000000001/000000010000000000000003

        full backup: 20210322-171211F
            timestamp start/stop: 2021-03-22 17:12:11 / 2021-03-22 17:12:28
            wal start/stop: 000000010000000000000002 / 000000010000000000000002
            database size: 23.4MB, database backup size: 23.4MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB
            database list: postgres (13359)

Json output will include the repository information and any error information. If no stanzas are found, then [invalid] will be set as the name:

[
   {
       "archive":[],
       "backup":[],
       "cipher":"none",
       "db":[],
       "name":"[invalid]",
       "repo":[
           {
               "cipher":"none",
               "key":1,
               "status":{
                   "code":99,
                   "message":"[PathOpenError] unable to list file info for path '/var/lib/pgbackrest/repo2/backup': [13] Permission denied"
               }
           }
       ],
       "status":{
           "code":99,
           "lock":{"backup":{"held":false}},
           "message":"other"
           }
   }
]
This commit is contained in:
Cynthia Shang
2021-03-25 12:29:36 -04:00
committed by GitHub
parent 7d7ac0e0eb
commit 2789d3b620
5 changed files with 617 additions and 226 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/xml/reference.xml
View File

@ -1632,7 +1632,7 @@
For machine-readable output use <br-option>--output=json</br-option>. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
Each stanza has a separate section and it is possible to limit output to a single stanza with the <br-option>--stanza</br-option> option. The stanza '<id>status</id>' gives a brief indication of the stanza's health. If this is '<id>ok</id>' then <backrest/> is functioning normally. If there are multiple repositories, then a status of '<id>mixed</id>' indicates that the stanza is not in a healthy state on one or more of the repositories. In this case the state of the stanza will be detailed on additional lines per repository. The '<id>wal archive min/max</id>' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the <br-option>{[dash]}-repo</br-option> option is set. Note that there may be gaps due to archive retention policies or other reasons.
Each stanza has a separate section and it is possible to limit output to a single stanza with the <br-option>--stanza</br-option> option. The stanza '<id>status</id>' gives a brief indication of the stanza's health. If this is '<id>ok</id>' then <backrest/> is functioning normally. If there are multiple repositories, then a status of '<id>mixed</id>' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of '<id>other</id>' will be used and the full error details will be provided. The '<id>wal archive min/max</id>' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the <br-option>{[dash]}-repo</br-option> option is set. Note that there may be gaps due to archive retention policies or other reasons.
The '<id>backup/expire running</id>' message will appear beside the '<id>status</id>' information if one of those commands is currently running on the host.

1
doc/xml/release.xml
View File

@ -70,6 +70,7 @@
<commit subject="Add write fault-tolerance to archive-push command."/>
<commit subject="Make --repo optional for remaining commands except stanza-delete."/>
<commit subject="Allow stanza-* commands to be run remotely."/>
<commit subject="Improve info command fault tolerance."/>
<release-item-contributor-list>
<release-item-contributor id="cynthia.shang"/>

46
src/command/help/help.auto.c
View File

@ -247,7 +247,7 @@ static const unsigned char helpDataPack[] =
pckTypeStr << 4 | 0x09, 0x23, // Summary
0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x20, 0x69, 0x6E, 0x66, 0x6F, 0x72, 0x6D, 0x61, 0x74, 0x69, 0x6F, 0x6E,
0x20, 0x61, 0x62, 0x6F, 0x75, 0x74, 0x20, 0x62, 0x61, 0x63, 0x6B, 0x75, 0x70, 0x73, 0x2E,
pckTypeStr << 4 | 0x08, 0xA3, 0x15, // Description
pckTypeStr << 4 | 0x08, 0xC6, 0x16, // Description
0x54, 0x68, 0x65, 0x20, 0x69, 0x6E, 0x66, 0x6F, 0x20, 0x63, 0x6F, 0x6D, 0x6D, 0x61, 0x6E, 0x64, 0x20, 0x6F, 0x70, 0x65,
0x72, 0x61, 0x74, 0x65, 0x73, 0x20, 0x6F, 0x6E, 0x20, 0x61, 0x20, 0x73, 0x69, 0x6E, 0x67, 0x6C, 0x65, 0x20, 0x73, 0x74,
0x61, 0x6E, 0x7A, 0x61, 0x20, 0x6F, 0x72, 0x20, 0x61, 0x6C, 0x6C, 0x20, 0x73, 0x74, 0x61, 0x6E, 0x7A, 0x61, 0x73, 0x2E,
@ -286,26 +286,34 @@ static const unsigned char helpDataPack[] =
0x20, 0x73, 0x74, 0x61, 0x6E, 0x7A, 0x61, 0x20, 0x69, 0x73, 0x20, 0x6E, 0x6F, 0x74, 0x20, 0x69, 0x6E, 0x20, 0x61, 0x20,
0x68, 0x65, 0x61, 0x6C, 0x74, 0x68, 0x79, 0x20, 0x73, 0x74, 0x61, 0x74, 0x65, 0x20, 0x6F, 0x6E, 0x20, 0x6F, 0x6E, 0x65,
0x20, 0x6F, 0x72, 0x20, 0x6D, 0x6F, 0x72, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x74, 0x68, 0x65, 0x20, 0x72, 0x65, 0x70, 0x6F,
0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x2E, 0x20, 0x49, 0x6E, 0x20, 0x74, 0x68, 0x69, 0x73, 0x20, 0x63, 0x61,
0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x3B, 0x20, 0x69, 0x6E, 0x20, 0x74, 0x68, 0x69, 0x73, 0x20, 0x63, 0x61,
0x73, 0x65, 0x20, 0x74, 0x68, 0x65, 0x20, 0x73, 0x74, 0x61, 0x74, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x74, 0x68, 0x65, 0x20,
0x73, 0x74, 0x61, 0x6E, 0x7A, 0x61, 0x20, 0x77, 0x69, 0x6C, 0x6C, 0x20, 0x62, 0x65, 0x20, 0x64, 0x65, 0x74, 0x61, 0x69,
0x6C, 0x65, 0x64, 0x20, 0x6F, 0x6E, 0x20, 0x61, 0x64, 0x64, 0x69, 0x74, 0x69, 0x6F, 0x6E, 0x61, 0x6C, 0x20, 0x6C, 0x69,
0x6E, 0x65, 0x73, 0x20, 0x70, 0x65, 0x72, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x79, 0x2E, 0x20,
0x54, 0x68, 0x65, 0x20, 0x27, 0x77, 0x61, 0x6C, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20, 0x6D, 0x69, 0x6E,
0x2F, 0x6D, 0x61, 0x78, 0x27, 0x20, 0x73, 0x68, 0x6F, 0x77, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x6D, 0x69, 0x6E, 0x69,
0x6D, 0x75, 0x6D, 0x20, 0x61, 0x6E, 0x64, 0x20, 0x6D, 0x61, 0x78, 0x69, 0x6D, 0x75, 0x6D, 0x20, 0x57, 0x41, 0x4C, 0x20,
0x63, 0x75, 0x72, 0x72, 0x65, 0x6E, 0x74, 0x6C, 0x79, 0x20, 0x73, 0x74, 0x6F, 0x72, 0x65, 0x64, 0x20, 0x69, 0x6E, 0x20,
0x74, 0x68, 0x65, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20, 0x61, 0x6E, 0x64, 0x2C, 0x20, 0x69, 0x6E, 0x20,
0x74, 0x68, 0x65, 0x20, 0x63, 0x61, 0x73, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x6D, 0x75, 0x6C, 0x74, 0x69, 0x70, 0x6C, 0x65,
0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x2C, 0x20, 0x77, 0x69, 0x6C, 0x6C, 0x20,
0x62, 0x65, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x72, 0x74, 0x65, 0x64, 0x20, 0x61, 0x63, 0x72, 0x6F, 0x73, 0x73, 0x20, 0x61,
0x6C, 0x6C, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x20, 0x75, 0x6E, 0x6C, 0x65,
0x73, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x2D, 0x2D, 0x72, 0x65, 0x70, 0x6F, 0x20, 0x6F, 0x70, 0x74, 0x69, 0x6F, 0x6E,
0x20, 0x69, 0x73, 0x20, 0x73, 0x65, 0x74, 0x2E, 0x20, 0x4E, 0x6F, 0x74, 0x65, 0x20, 0x74, 0x68, 0x61, 0x74, 0x20, 0x74,
0x68, 0x65, 0x72, 0x65, 0x20, 0x6D, 0x61, 0x79, 0x20, 0x62, 0x65, 0x20, 0x67, 0x61, 0x70, 0x73, 0x20, 0x64, 0x75, 0x65,
0x20, 0x74, 0x6F, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20, 0x72, 0x65, 0x74, 0x65, 0x6E, 0x74, 0x69, 0x6F,
0x6E, 0x20, 0x70, 0x6F, 0x6C, 0x69, 0x63, 0x69, 0x65, 0x73, 0x20, 0x6F, 0x72, 0x20, 0x6F, 0x74, 0x68, 0x65, 0x72, 0x20,
0x72, 0x65, 0x61, 0x73, 0x6F, 0x6E, 0x73, 0x2E, 0x0A, 0x0A,
0x6C, 0x65, 0x64, 0x20, 0x70, 0x65, 0x72, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x79, 0x2E, 0x20,
0x46, 0x6F, 0x72, 0x20, 0x63, 0x61, 0x73, 0x65, 0x73, 0x20, 0x69, 0x6E, 0x20, 0x77, 0x68, 0x69, 0x63, 0x68, 0x20, 0x61,
0x6E, 0x20, 0x65, 0x72, 0x72, 0x6F, 0x72, 0x20, 0x6F, 0x6E, 0x20, 0x61, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74,
0x6F, 0x72, 0x79, 0x20, 0x6F, 0x63, 0x63, 0x75, 0x72, 0x72, 0x65, 0x64, 0x20, 0x74, 0x68, 0x61, 0x74, 0x20, 0x69, 0x73,
0x20, 0x6E, 0x6F, 0x74, 0x20, 0x6F, 0x6E, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x74, 0x68, 0x65, 0x20, 0x6B, 0x6E, 0x6F, 0x77,
0x6E, 0x20, 0x65, 0x72, 0x72, 0x6F, 0x72, 0x20, 0x63, 0x6F, 0x64, 0x65, 0x73, 0x2C, 0x20, 0x74, 0x68, 0x65, 0x6E, 0x20,
0x61, 0x6E, 0x20, 0x65, 0x72, 0x72, 0x6F, 0x72, 0x20, 0x63, 0x6F, 0x64, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x27, 0x6F, 0x74,
0x68, 0x65, 0x72, 0x27, 0x20, 0x77, 0x69, 0x6C, 0x6C, 0x20, 0x62, 0x65, 0x20, 0x75, 0x73, 0x65, 0x64, 0x20, 0x61, 0x6E,
0x64, 0x20, 0x74, 0x68, 0x65, 0x20, 0x66, 0x75, 0x6C, 0x6C, 0x20, 0x65, 0x72, 0x72, 0x6F, 0x72, 0x20, 0x64, 0x65, 0x74,
0x61, 0x69, 0x6C, 0x73, 0x20, 0x77, 0x69, 0x6C, 0x6C, 0x20, 0x62, 0x65, 0x20, 0x70, 0x72, 0x6F, 0x76, 0x69, 0x64, 0x65,
0x64, 0x2E, 0x20, 0x54, 0x68, 0x65, 0x20, 0x27, 0x77, 0x61, 0x6C, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20,
0x6D, 0x69, 0x6E, 0x2F, 0x6D, 0x61, 0x78, 0x27, 0x20, 0x73, 0x68, 0x6F, 0x77, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x6D,
0x69, 0x6E, 0x69, 0x6D, 0x75, 0x6D, 0x20, 0x61, 0x6E, 0x64, 0x20, 0x6D, 0x61, 0x78, 0x69, 0x6D, 0x75, 0x6D, 0x20, 0x57,
0x41, 0x4C, 0x20, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6E, 0x74, 0x6C, 0x79, 0x20, 0x73, 0x74, 0x6F, 0x72, 0x65, 0x64, 0x20,
0x69, 0x6E, 0x20, 0x74, 0x68, 0x65, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20, 0x61, 0x6E, 0x64, 0x2C, 0x20,
0x69, 0x6E, 0x20, 0x74, 0x68, 0x65, 0x20, 0x63, 0x61, 0x73, 0x65, 0x20, 0x6F, 0x66, 0x20, 0x6D, 0x75, 0x6C, 0x74, 0x69,
0x70, 0x6C, 0x65, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x2C, 0x20, 0x77, 0x69,
0x6C, 0x6C, 0x20, 0x62, 0x65, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x72, 0x74, 0x65, 0x64, 0x20, 0x61, 0x63, 0x72, 0x6F, 0x73,
0x73, 0x20, 0x61, 0x6C, 0x6C, 0x20, 0x72, 0x65, 0x70, 0x6F, 0x73, 0x69, 0x74, 0x6F, 0x72, 0x69, 0x65, 0x73, 0x20, 0x75,
0x6E, 0x6C, 0x65, 0x73, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x2D, 0x2D, 0x72, 0x65, 0x70, 0x6F, 0x20, 0x6F, 0x70, 0x74,
0x69, 0x6F, 0x6E, 0x20, 0x69, 0x73, 0x20, 0x73, 0x65, 0x74, 0x2E, 0x20, 0x4E, 0x6F, 0x74, 0x65, 0x20, 0x74, 0x68, 0x61,
0x74, 0x20, 0x74, 0x68, 0x65, 0x72, 0x65, 0x20, 0x6D, 0x61, 0x79, 0x20, 0x62, 0x65, 0x20, 0x67, 0x61, 0x70, 0x73, 0x20,
0x64, 0x75, 0x65, 0x20, 0x74, 0x6F, 0x20, 0x61, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x20, 0x72, 0x65, 0x74, 0x65, 0x6E,
0x74, 0x69, 0x6F, 0x6E, 0x20, 0x70, 0x6F, 0x6C, 0x69, 0x63, 0x69, 0x65, 0x73, 0x20, 0x6F, 0x72, 0x20, 0x6F, 0x74, 0x68,
0x65, 0x72, 0x20, 0x72, 0x65, 0x61, 0x73, 0x6F, 0x6E, 0x73, 0x2E, 0x0A, 0x0A,
0x54, 0x68, 0x65, 0x20, 0x27, 0x62, 0x61, 0x63, 0x6B, 0x75, 0x70, 0x2F, 0x65, 0x78, 0x70, 0x69, 0x72, 0x65, 0x20, 0x72,
0x75, 0x6E, 0x6E, 0x69, 0x6E, 0x67, 0x27, 0x20, 0x6D, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x20, 0x77, 0x69, 0x6C, 0x6C,
0x20, 0x61, 0x70, 0x70, 0x65, 0x61, 0x72, 0x20, 0x62, 0x65, 0x73, 0x69, 0x64, 0x65, 0x20, 0x74, 0x68, 0x65, 0x20, 0x27,

300
src/command/info/info.c
View File

@ -90,6 +90,12 @@ STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR, "missing sta
STRING_STATIC(INFO_STANZA_MESSAGE_MIXED_STR, "different across repos");
#define INFO_STANZA_STATUS_CODE_PG_MISMATCH 5
STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR, "database mismatch across repos");
#define INFO_STANZA_STATUS_CODE_BACKUP_MISSING 6
STRING_STATIC(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR, "requested backup not found");
#define INFO_STANZA_STATUS_CODE_OTHER 99
#define INFO_STANZA_STATUS_MESSAGE_OTHER "other"
STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_OTHER_STR, INFO_STANZA_STATUS_MESSAGE_OTHER);
STRING_STATIC(INFO_STANZA_INVALID_STR, "[invalid]");
#define INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "backup/expire running"
@ -106,6 +112,8 @@ typedef struct InfoRepoData
unsigned int backupIdx; // Index of the next backup that may be a candidate for sorting
InfoBackup *backupInfo; // Contents of the backup.info file of the stanza on this repo
InfoArchive *archiveInfo; // Contents of the archive.info file of the stanza on this repo
Manifest *manifest; // Contents of manifest if backup requested and is on this repo
String *error; // Formatted error
} InfoRepoData;
#define FUNCTION_LOG_INFO_REPO_DATA_TYPE \
@ -119,6 +127,8 @@ typedef struct InfoStanzaRepo
const String *name; // Name of the stanza
uint64_t currentPgSystemId; // Current postgres system id for the stanza
unsigned int currentPgVersion; // Current postgres version for the stanza
bool backupLockChecked; // Has the check for a backup lock already been performed?
bool backupLockHeld; // Is backup lock held on the system where info command is run?
InfoRepoData *repoList; // List of configured repositories
} InfoStanzaRepo;
@ -143,6 +153,24 @@ typedef struct DbGroup
#define FUNCTION_LOG_DB_GROUP_FORMAT(value, buffer, bufferSize) \
objToLog(value, "DbGroup", buffer, bufferSize)
/***********************************************************************************************************************************
Helper function for reporting errors
***********************************************************************************************************************************/
static void
infoStanzaErrorAdd(InfoRepoData *repoList, const ErrorType *type, const String *message)
{
repoList->stanzaStatus = INFO_STANZA_STATUS_CODE_OTHER;
repoList->error = strNewFmt("[%s] %s", errorTypeName(type), strZ(message));
// Free the info objects for this stanza since we cannot process it
infoBackupFree(repoList->backupInfo);
infoArchiveFree(repoList->archiveInfo);
manifestFree(repoList->manifest);
repoList->backupInfo = NULL;
repoList->archiveInfo = NULL;
repoList->manifest = NULL;
}
/***********************************************************************************************************************************
Set the overall error status code and message for the stanza to the code and message passed
***********************************************************************************************************************************/
@ -155,7 +183,7 @@ stanzaStatus(const int code, bool backupLockHeld, Variant *stanzaInfo)
FUNCTION_TEST_PARAM(VARIANT, stanzaInfo);
FUNCTION_TEST_END();
ASSERT(code >= 0 && code <= 5);
ASSERT((code >= 0 && code <= 6) || code == 99);
ASSERT(stanzaInfo != NULL);
KeyValue *statusKv = kvPutKv(varKv(stanzaInfo), STANZA_KEY_STATUS_VAR);
@ -187,6 +215,14 @@ stanzaStatus(const int code, bool backupLockHeld, Variant *stanzaInfo)
case INFO_STANZA_STATUS_CODE_PG_MISMATCH:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR));
break;
case INFO_STANZA_STATUS_CODE_BACKUP_MISSING:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR));
break;
case INFO_STANZA_STATUS_CODE_OTHER:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OTHER_STR));
break;
}
// Construct a specific lock part
@ -201,14 +237,15 @@ stanzaStatus(const int code, bool backupLockHeld, Variant *stanzaInfo)
Set the error status code and message for the stanza on the repo to the code and message passed
***********************************************************************************************************************************/
static void
repoStanzaStatus(const int code, Variant *repoStanzaInfo)
repoStanzaStatus(const int code, Variant *repoStanzaInfo, InfoRepoData *repoData)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM(INT, code);
FUNCTION_TEST_PARAM(VARIANT, repoStanzaInfo);
FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData);
FUNCTION_TEST_END();
ASSERT(code >= 0 && code <= 3);
ASSERT((code >= 0 && code <= 3) || code == 99 || code == 6);
ASSERT(repoStanzaInfo != NULL);
KeyValue *statusKv = kvPutKv(varKv(repoStanzaInfo), STANZA_KEY_STATUS_VAR);
@ -232,6 +269,14 @@ repoStanzaStatus(const int code, Variant *repoStanzaInfo)
case INFO_STANZA_STATUS_CODE_NO_BACKUP:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR));
break;
case INFO_STANZA_STATUS_CODE_BACKUP_MISSING:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR));
break;
case INFO_STANZA_STATUS_CODE_OTHER:
kvAdd(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(repoData->error));
break;
}
FUNCTION_TEST_RETURN_VOID();
@ -336,14 +381,13 @@ Add the backup data to the backup section
***********************************************************************************************************************************/
static void
backupListAdd(
VariantList *backupSection, InfoBackupData *backupData, const String *backupLabel, InfoRepoData *repoData, unsigned int repoIdx)
VariantList *backupSection, InfoBackupData *backupData, const String *backupLabel, InfoRepoData *repoData)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM(VARIANT_LIST, backupSection); // The section to add the backup data to
FUNCTION_TEST_PARAM_P(INFO_BACKUP_DATA, backupData); // The data for the backup
FUNCTION_TEST_PARAM(STRING, backupLabel); // Backup label to filter if requested by the user
FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData); // The repo data where this backup is located
FUNCTION_TEST_PARAM(UINT, repoIdx); // Internal index for the repo
FUNCTION_TEST_END();
ASSERT(backupSection != NULL);
@ -403,20 +447,15 @@ backupListAdd(
kvAdd(timeInfo, KEY_START_VAR, VARUINT64((uint64_t)backupData->backupTimestampStart));
kvAdd(timeInfo, KEY_STOP_VAR, VARUINT64((uint64_t)backupData->backupTimestampStop));
// If a backup label was specified and this is that label, then get the manifest
// If a backup label was specified and this is that label, then get the data from the loaded manifest
if (backupLabel != NULL && strEq(backupData->backupLabel, backupLabel))
{
// Load the manifest file
Manifest *manifest = manifestLoadFile(
storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)),
repoData->cipher, infoPgCipherPass(infoBackupPg(repoData->backupInfo)));
// Get the list of databases in this backup
VariantList *databaseSection = varLstNew();
for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(manifest); dbIdx++)
for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(repoData->manifest); dbIdx++)
{
const ManifestDb *db = manifestDb(manifest, dbIdx);
const ManifestDb *db = manifestDb(repoData->manifest, dbIdx);
// Do not display template databases
if (db->id > db->lastSystemId)
@ -436,9 +475,9 @@ backupListAdd(
VariantList *linkSection = varLstNew();
VariantList *tablespaceSection = varLstNew();
for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++)
for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(repoData->manifest); targetIdx++)
{
const ManifestTarget *target = manifestTarget(manifest, targetIdx);
const ManifestTarget *target = manifestTarget(repoData->manifest, targetIdx);
Variant *link = varNewKv(kvNew());
Variant *tablespace = varNewKv(kvNew());
@ -476,9 +515,9 @@ backupListAdd(
// Get the list of files with an error in the page checksum
VariantList *checksumPageErrorList = varLstNew();
for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++)
for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(repoData->manifest); fileIdx++)
{
const ManifestFile *file = manifestFile(manifest, fileIdx);
const ManifestFile *file = manifestFile(repoData->manifest, fileIdx);
if (file->checksumPageError)
varLstAdd(checksumPageErrorList, varNewStr(manifestPathPg(file->name)));
@ -488,7 +527,8 @@ backupListAdd(
varKv(backupInfo), BACKUP_KEY_CHECKSUM_PAGE_ERROR_VAR,
(!varLstEmpty(checksumPageErrorList) ? varNewVarLst(checksumPageErrorList) : NULL));
manifestFree(manifest);
manifestFree(repoData->manifest);
repoData->manifest = NULL;
}
varLstAdd(backupSection, backupInfo);
@ -559,7 +599,7 @@ backupList(
repoData->backupIdx++;
// Add the backup data to the backup section
backupListAdd(backupSection, &backupData, backupLabel, repoData, backupNextRepoIdx);
backupListAdd(backupSection, &backupData, backupLabel, repoData);
backupTotalProcessed++;
}
@ -601,7 +641,6 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
int stanzaStatusCode = -1;
unsigned int stanzaCipherType = 0;
bool checkBackupLock = false;
// Set the stanza name and initialize the overall stanza variables
kvPut(varKv(stanzaInfo), KEY_NAME_VAR, VARSTR(stanzaData->name));
@ -620,10 +659,13 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && repoData->backupInfo == NULL)
repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH;
TRY_BEGIN()
{
// If the backup.info file has been read, then get the backup and archive information on this repo
if (repoData->backupInfo != NULL)
{
// If the backup.info file exists, get the database history information (oldest to newest) and corresponding archive
// If the backup.info file exists, get the database history information (oldest to newest) and corresponding
// archive
for (unsigned int pgIdx = infoPgDataTotal(infoBackupPg(repoData->backupInfo)) - 1; (int)pgIdx >= 0; pgIdx--)
{
InfoPgData pgData = infoPgData(infoBackupPg(repoData->backupInfo), pgIdx);
@ -638,8 +680,8 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
// Get the archive info for the DB from the archive.info file
archiveDbList(
stanzaData->name, &pgData, archiveSection, repoData->archiveInfo, (pgIdx == 0 ? true : false), repoIdx,
repoData->key);
stanzaData->name, &pgData, archiveSection, repoData->archiveInfo, (pgIdx == 0 ? true : false),
repoIdx, repoData->key);
}
// Set stanza status if the current db sections do not match across repos
@ -647,23 +689,24 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
infoBackupPg(repoData->backupInfo), infoPgDataCurrentId(infoBackupPg(repoData->backupInfo)));
// The current PG system and version must match across repos for the stanza, if not, a failure may have occurred
// during an upgrade or the repo may have been disabled during the stanza upgrade to protect from error propagation
// during an upgrade or the repo may have been disabled during the stanza upgrade to protect from error
// propagation
if (stanzaData->currentPgVersion != backupInfoCurrentPg.version ||
stanzaData->currentPgSystemId != backupInfoCurrentPg.systemId)
{
stanzaStatusCode = INFO_STANZA_STATUS_CODE_PG_MISMATCH;
}
}
// If the stanza has been created successfully on at least one repo, then check for a lock on the PG server
if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK)
}
CATCH_ANY()
{
checkBackupLock = true;
infoStanzaErrorAdd(repoData, errorType(), STR(errorMessage()));
}
TRY_END();
// If there are no current backups on this repo then set status to no backup
if (infoBackupDataTotal(repoData->backupInfo) == 0)
if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && infoBackupDataTotal(repoData->backupInfo) == 0)
repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_NO_BACKUP;
}
// Track the status over all repos if the status for the stanza has not already been determined
if (stanzaStatusCode != INFO_STANZA_STATUS_CODE_PG_MISMATCH)
@ -684,7 +727,7 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
stanzaCipherType = stanzaCipherType != repoData->cipher ? INFO_STANZA_STATUS_CODE_MIXED : repoData->cipher;
// Add the status of the stanza on the repo to the repo section, and the repo to the repo array
repoStanzaStatus(repoData->stanzaStatus, repoInfo);
repoStanzaStatus(repoData->stanzaStatus, repoInfo, repoData);
varLstAdd(repoSection, repoInfo);
// Add the database history, backup, archive and repo arrays to the stanza info
@ -697,21 +740,8 @@ stanzaInfoList(List *stanzaRepoList, const String *backupLabel, unsigned int rep
backupList(backupSection, stanzaData, backupLabel, repoIdxMin, repoIdxMax);
kvPut(varKv(stanzaInfo), STANZA_KEY_BACKUP_VAR, varNewVarLst(backupSection));
static bool backupLockHeld = false;
// If the stanza is OK on at least one repo, then check if there's a local backup running
if (checkBackupLock)
{
// Try to acquire a lock. If not possible, assume another backup or expire is already running.
backupLockHeld = !lockAcquire(
cfgOptionStr(cfgOptLockPath), stanzaData->name, cfgOptionStr(cfgOptExecId), lockTypeBackup, 0, false);
// Immediately release the lock acquired
lockRelease(!backupLockHeld);
}
// Set the overall stanza status
stanzaStatus(stanzaStatusCode, backupLockHeld, stanzaInfo);
stanzaStatus(stanzaStatusCode, stanzaData->backupLockHeld, stanzaInfo);
// Set the overall cipher type
if (stanzaCipherType != INFO_STANZA_STATUS_CODE_MIXED)
@ -1053,30 +1083,32 @@ formatTextDb(
Get the backup and archive info files on the specified repo for the stanza
***********************************************************************************************************************************/
static void
infoUpdateStanza(const Storage *storage, InfoStanzaRepo *stanzaRepo, unsigned int repoIdx, bool stanzaExists)
infoUpdateStanza(
const Storage *storage, InfoStanzaRepo *stanzaRepo, unsigned int repoIdx, bool stanzaExists, const String *backupLabel)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM(STORAGE, storage);
FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaRepo);
FUNCTION_TEST_PARAM(UINT, repoIdx);
FUNCTION_TEST_PARAM(BOOL, stanzaExists);
FUNCTION_TEST_PARAM(STRING, backupLabel);
FUNCTION_TEST_END();
ASSERT(storage != NULL);
ASSERT(stanzaRepo != NULL);
InfoBackup *info = NULL;
volatile int stanzaStatus = INFO_STANZA_STATUS_CODE_OK;
// If the stanza exists, attempt to get the backup.info file
// If the stanza exists, attempt to get the info files
if (stanzaExists)
{
TRY_BEGIN()
{
// Catch certain errors
TRY_BEGIN()
{
// Attempt to load the backup info file
info = infoBackupLoadFile(
stanzaRepo->repoList[repoIdx].backupInfo = infoBackupLoadFile(
storage, strNewFmt(STORAGE_PATH_BACKUP "/%s/%s", strZ(stanzaRepo->name), INFO_BACKUP_FILE),
stanzaRepo->repoList[repoIdx].cipher, stanzaRepo->repoList[repoIdx].cipherPass);
}
@ -1096,21 +1128,47 @@ infoUpdateStanza(const Storage *storage, InfoStanzaRepo *stanzaRepo, unsigned in
}
TRY_END();
// If backup.info was found, then get the archive.info file, which must exist if the backup.info exists, else throw error
if (info != NULL)
// If backup.info was found, then get the archive.info file, which must exist if the backup.info exists, else the failed
// load will throw an error which will be trapped and recorded
if (stanzaRepo->repoList[repoIdx].backupInfo != NULL)
{
stanzaRepo->repoList[repoIdx].archiveInfo = infoArchiveLoadFile(
storage, strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanzaRepo->name), INFO_ARCHIVE_FILE),
stanzaRepo->repoList[repoIdx].cipher, stanzaRepo->repoList[repoIdx].cipherPass);
// If a specific backup exists on this repo then attempt to load the manifest
if (backupLabel != NULL)
{
stanzaRepo->repoList[repoIdx].manifest = manifestLoadFile(
storage, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)),
stanzaRepo->repoList[repoIdx].cipher,
infoPgCipherPass(infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo)));
}
// If a backup lock check has not already been performed, then do so
if (!stanzaRepo->backupLockChecked)
{
// Try to acquire a lock. If not possible, assume another backup or expire is already running.
stanzaRepo->backupLockHeld = !lockAcquire(
cfgOptionStr(cfgOptLockPath), stanzaRepo->name, cfgOptionStr(cfgOptExecId), lockTypeBackup, 0, false);
// Immediately release the lock acquired
lockRelease(!stanzaRepo->backupLockHeld);
stanzaRepo->backupLockChecked = true;
}
}
stanzaRepo->repoList[repoIdx].stanzaStatus = stanzaStatus;
}
CATCH_ANY()
{
infoStanzaErrorAdd(&stanzaRepo->repoList[repoIdx], errorType(), STR(errorMessage()));
}
TRY_END();
}
else
stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH;
stanzaRepo->repoList[repoIdx].backupInfo = info;
// If the backup.info and therefore archive.info exist, and the currentPg has not been set for the stanza, then set it
if (stanzaRepo->currentPgVersion == 0 && stanzaRepo->repoList[repoIdx].backupInfo != NULL)
{
@ -1164,7 +1222,26 @@ infoRender(void)
repoIdxMax = repoIdxMin;
}
// Initialize error reporting at the repo level
InfoRepoData *repoErrorList = memNew(repoTotal * sizeof(InfoRepoData));
bool repoError = false;
for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++)
{
// Initialize the error list on this repo
repoErrorList[repoIdx] = (InfoRepoData)
{
.key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx),
.cipher = cipherType(cfgOptionIdxStr(cfgOptRepoCipherType, repoIdx)),
.cipherPass = cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx),
.error = NULL,
};
// Initialize backup label indicator
const String *backupExistsOnRepo = NULL;
// Catch any repo errors
TRY_BEGIN()
{
// Get the repo storage in case it is remote and encryption settings need to be pulled down
const Storage *storageRepo = storageRepoIdx(repoIdx);
@ -1172,8 +1249,13 @@ infoRender(void)
// If a backup set was specified, see if the manifest exists
if (backupLabel != NULL)
{
// If the backup exists on this repo, set the global indicator that we found it on at least one repo and
// set the exists label for later loading of the manifest
if (storageExistsP(storageRepo, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel))))
{
backupFound = true;
backupExistsOnRepo = backupLabel;
}
}
// Get a list of stanzas in the backup directory
@ -1202,10 +1284,10 @@ infoRender(void)
// Get the stanza if it is already in the list
InfoStanzaRepo *stanzaRepo = lstFind(stanzaRepoList, &stanzaName);
// If the stanza was already added to the array, then update this repo for the stanza, else the stanza has not yet
// been added to the list, so add it
// If the stanza was already added to the array, then update this repo for the stanza, else the stanza has not
// yet been added to the list, so add it
if (stanzaRepo != NULL)
infoUpdateStanza(storageRepo, stanzaRepo, repoIdx, stanzaExists);
infoUpdateStanza(storageRepo, stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo);
else
{
InfoStanzaRepo stanzaRepo =
@ -1224,27 +1306,82 @@ infoRender(void)
.key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoListIdx),
.cipher = cipherType(cfgOptionIdxStr(cfgOptRepoCipherType, repoListIdx)),
.cipherPass = cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoListIdx),
.error = NULL,
};
}
// Update the info for this repo
infoUpdateStanza(storageRepo, &stanzaRepo, repoIdx, stanzaExists);
infoUpdateStanza(storageRepo, &stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo);
lstAdd(stanzaRepoList, &stanzaRepo);
}
}
}
// If a backup label was requested but it was not found on any repo
if (backupLabel != NULL && !backupFound)
CATCH_ANY()
{
THROW_FMT(
FileMissingError, "manifest does not exist for backup '%s'\n"
"HINT: is the backup listed when running the info command with --stanza option only?", strZ(backupLabel));
// At this point, stanza-level errors were caught and stored in the stanza structure, any errors caught here are due
// to higher level problems (e.g. repo inaccessible, invalid permissions on the repo, etc) and will be reported
// later if there are valid stanzas. If there are no valid stanzas after the loop exits, then these errors will be
// reported with a stanza named "[invalid]".
infoStanzaErrorAdd(&repoErrorList[repoIdx], errorType(), STR(errorMessage()));
repoError = true;
}
TRY_END();
}
VariantList *infoList = varLstNew();
String *resultStr = strNew("");
// Record any repository-level errors with each stanza -- if there are no stanzas and one was not requested, then create an
// "[invalid]" one for reporting
if (repoError)
{
if (lstEmpty(stanzaRepoList))
{
InfoStanzaRepo stanzaRepo =
{
.name = stanza != NULL ? stanza : INFO_STANZA_INVALID_STR,
.currentPgVersion = 0,
.currentPgSystemId = 0,
.repoList = repoErrorList,
};
lstAdd(stanzaRepoList, &stanzaRepo);
}
else
{
// For each stanza report the repos that were in error
for (unsigned int idx = 0; idx < lstSize(stanzaRepoList); idx++)
{
InfoStanzaRepo *stanzaData = lstGet(stanzaRepoList, idx);
for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++)
{
if (repoErrorList[repoIdx].error != NULL)
stanzaData->repoList[repoIdx] = repoErrorList[repoIdx];
}
}
}
}
// If a backup label was requested but it was not found on any repo, report the error here rather than individually to avoid
// listing each repo as "requested backup not found"
if (backupLabel != NULL && !backupFound)
{
// Get the stanza record and update each repo to indicate backup not found where there is not already an error status
// so that errors on other repositories will be displayed and not overwritten
InfoStanzaRepo *stanzaRepo = lstFind(stanzaRepoList, &stanza);
for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++)
{
if (stanzaRepo->repoList[repoIdx].stanzaStatus == INFO_STANZA_STATUS_CODE_OK)
{
stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_BACKUP_MISSING;
infoBackupFree(stanzaRepo->repoList[repoIdx].backupInfo);
stanzaRepo->repoList[repoIdx].backupInfo = NULL;
}
}
}
// If the backup storage exists, then search for and process any stanzas
if (!lstEmpty(stanzaRepoList))
infoList = stanzaInfoList(stanzaRepoList, backupLabel, repoIdxMin, repoIdxMax);
@ -1279,8 +1416,10 @@ infoRender(void)
if (statusCode != INFO_STANZA_STATUS_CODE_OK)
{
// Update the overall stanza status and change displayed status if backup lock is found
if (statusCode == INFO_STANZA_STATUS_CODE_MIXED || statusCode == INFO_STANZA_STATUS_CODE_PG_MISMATCH)
if (statusCode == INFO_STANZA_STATUS_CODE_MIXED || statusCode == INFO_STANZA_STATUS_CODE_PG_MISMATCH ||
statusCode == INFO_STANZA_STATUS_CODE_OTHER)
{
// Stanza status
strCatFmt(
resultStr, "%s%s\n",
statusCode == INFO_STANZA_STATUS_CODE_MIXED ? INFO_STANZA_MIXED :
@ -1290,19 +1429,40 @@ infoRender(void)
// Output the status per repo
VariantList *repoSection = kvGetList(stanzaInfo, STANZA_KEY_REPO_VAR);
bool multiRepo = varLstSize(repoSection) > 1;
const char *formatSpacer = multiRepo ? " " : " ";
for (unsigned int repoIdx = 0; repoIdx < varLstSize(repoSection); repoIdx++)
{
KeyValue *repoInfo = varKv(varLstGet(repoSection, repoIdx));
KeyValue *repoStatus = varKv(kvGet(repoInfo, STANZA_KEY_STATUS_VAR));
// If more than one repo configured, then add the repo status per repo
if (multiRepo)
strCatFmt(resultStr, " repo%u: ", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR)));
if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OK)
strCatZ(resultStr, INFO_STANZA_STATUS_OK "\n");
else
{
if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OTHER)
{
StringList *repoError = strLstNewSplit(
varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR)), STRDEF("\n"));
strCatFmt(
resultStr, " repo%u: ", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR)));
resultStr, "%s%s%s\n",
multiRepo ? INFO_STANZA_STATUS_ERROR " (" INFO_STANZA_STATUS_MESSAGE_OTHER ")\n" : "",
formatSpacer, strZ(strLstJoin(repoError, strZ(strNewFmt("\n%s", formatSpacer)))));
}
else
{
strCatFmt(
resultStr, "%s",
varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OK ?
INFO_STANZA_STATUS_OK "\n" : strZ(strNewFmt(INFO_STANZA_STATUS_ERROR " (%s)\n",
strZ(varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR))))));
resultStr, INFO_STANZA_STATUS_ERROR " (%s)\n",
strZ(varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR))));
}
}
}
}
else

290
test/src/module/command/infoTest.c
View File

@ -154,17 +154,57 @@ testRun(void)
harnessInfoChecksum(content)),
"put backup info to file");
TEST_ERROR_FMT(infoRender(), FileMissingError,
"unable to load info file '%s/archive.info' or '%s/archive.info.copy':\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.",
strZ(archiveStanza1Path), strZ(archiveStanza1Path),
strZ(strNewFmt("%s/archive.info", strZ(archiveStanza1Path))),
strZ(strNewFmt("%s/archive.info.copy", strZ(archiveStanza1Path))));
TEST_RESULT_STR(
infoRender(), strNewFmt(
"["
"{"
"\"archive\":[],"
"\"backup\":[],"
"\"cipher\":\"none\","
"\"db\":[],"
"\"name\":\"stanza1\","
"\"repo\":["
"{"
"\"cipher\":\"none\","
"\"key\":1,"
"\"status\":{"
"\"code\":99,"
"\"message\":\"[FileMissingError] unable to load info file '%s/archive.info' or"
" '%s/archive.info.copy':\\n"
"FileMissingError: unable to open missing file '%s/archive.info' for read\\n"
"FileMissingError: unable to open missing file '%s/archive.info.copy' for read\\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\\n"
"HINT: is archive_command configured correctly in postgresql.conf?\\n"
"HINT: has a stanza-create been performed?\\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate"
" archiving scheme.\""
"}"
"}"
"],"
"\"status\":{"
"\"code\":99,"
"\"lock\":{\"backup\":{\"held\":false}},"
"\"message\":\"other\""
"}"
"}"
"]", strZ(archiveStanza1Path), strZ(archiveStanza1Path), strZ(archiveStanza1Path), strZ(archiveStanza1Path)),
"json - other error, single repo");
harnessCfgLoad(cfgCmdInfo, argListTextStanzaOpt);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza1\n"
" status: error (other)\n"
" [FileMissingError] unable to load info file '%s/stanza1/archive.info' or '%s/stanza1/archive.info.copy':\n"
" FileMissingError: unable to open missing file '%s/stanza1/archive.info' for read\n"
" FileMissingError: unable to open missing file '%s/stanza1/archive.info.copy' for read\n"
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.\n"
" cipher: none\n", strZ(archivePath), strZ(archivePath), strZ(archivePath), strZ(archivePath)),
"text - other error, single repo");
// backup.info/archive.info files exist, mismatched db ids, no backup:current section so no valid backups
// Only the current db information from the db:history will be processed.
@ -1566,6 +1606,22 @@ testRun(void)
}
HARNESS_FORK_END();
// Stanza exists but set requested backup does not
//--------------------------------------------------------------------------------------------------------------------------
argList2 = strLstDup(argListMultiRepo);
strLstAddZ(argList2, "--stanza=stanza1");
strLstAddZ(argList2, "--set=bogus");
harnessCfgLoad(cfgCmdInfo, argList2);
TEST_RESULT_STR_Z(
infoRender(),
"stanza: stanza1\n"
" status: error (requested backup not found)\n"
" cipher: mixed\n"
" repo1: none\n"
" repo2: aes-256-cbc\n",
"text, multi-repo, backup not found");
// Backup set requested, with 1 checksum error
//--------------------------------------------------------------------------------------------------------------------------
argList2 = strLstDup(argListMultiRepo);
@ -2293,6 +2349,7 @@ testRun(void)
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("encryption error");
// Change repo1 to have the same cipher type as repo2 even though on disk it does not
content = strNew
(
"[global]\n"
@ -2302,19 +2359,123 @@ testRun(void)
TEST_RESULT_VOID(
storagePutP(storageNewWriteP(storageLocalWrite(), strNewFmt("%s/pgbackrest.conf", testPath())),
BUFSTR(content)), "put pgbackrest.conf file");
strLstAddZ(argListText, "--repo-cipher-type=aes-256-cbc");
strLstAdd(argListText, strNewFmt("--config=%s/pgbackrest.conf", testPath()));
harnessCfgLoad(cfgCmdInfo, argListText);
TEST_ERROR_FMT(
infoRender(), CryptoError,
"unable to load info file '%s/backup.info' or '%s/backup.info.copy':\n"
"CryptoError: cipher header invalid\n"
"HINT: is or was the repo encrypted?\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"HINT: backup.info cannot be opened and is required to perform a backup.\n"
"HINT: has a stanza-create been performed?\n"
"HINT: use option --stanza if encryption settings are different for the stanza than the global settings.",
strZ(backupStanza1Path), strZ(backupStanza1Path), strZ(strNewFmt("%s/backup.info.copy", strZ(backupStanza1Path))));
argList2 = strLstDup(argListMultiRepo);
strLstAddZ(argList2, "--repo-cipher-type=aes-256-cbc");
strLstAdd(argList2, strNewFmt("--config=%s/pgbackrest.conf", testPath()));
harnessCfgLoad(cfgCmdInfo, argList2);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza1\n"
" status: mixed\n"
" repo1: error (other)\n"
" [CryptoError] unable to load info file '%s/stanza1/backup.info' or '%s/stanza1/backup.info.copy':\n"
" CryptoError: cipher header invalid\n"
" HINT: is or was the repo encrypted?\n"
" FileMissingError: unable to open missing file '%s/stanza1/backup.info.copy' for read\n"
" HINT: backup.info cannot be opened and is required to perform a backup.\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use option --stanza if encryption settings are different for the stanza than the global"
" settings.\n"
" repo2: error (no valid backups)\n"
" cipher: aes-256-cbc\n"
"\n"
" db (current)\n"
" wal archive min/max (9.5): 000000010000000000000003/000000010000000000000004\n"
"\n"
"stanza: stanza2\n"
" status: mixed\n"
" repo1: error (other)\n"
" [CryptoError] unable to load info file '%s/stanza2/backup.info' or '%s/stanza2/backup.info.copy':\n"
" CryptoError: cipher header invalid\n"
" HINT: is or was the repo encrypted?\n"
" FileMissingError: unable to open missing file '%s/stanza2/backup.info.copy' for read\n"
" HINT: backup.info cannot be opened and is required to perform a backup.\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use option --stanza if encryption settings are different for the stanza than the global"
" settings.\n"
" repo2: error (missing stanza path)\n"
" cipher: aes-256-cbc\n"
"\n"
"stanza: stanza3\n"
" status: mixed\n"
" repo1: error (other)\n"
" [CryptoError] unable to load info file '%s/stanza3/backup.info' or '%s/stanza3/backup.info.copy':\n"
" CryptoError: cipher header invalid\n"
" HINT: is or was the repo encrypted?\n"
" FileMissingError: unable to open missing file '%s/stanza3/backup.info.copy' for read\n"
" HINT: backup.info cannot be opened and is required to perform a backup.\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use option --stanza if encryption settings are different for the stanza than the global"
" settings.\n"
" repo2: ok\n"
" cipher: aes-256-cbc\n"
"\n"
" db (current)\n"
" wal archive min/max (9.4): 000000010000000000000001/000000010000000000000002\n"
"\n"
" full backup: 20201110-100000F\n"
" timestamp start/stop: 2020-11-10 10:00:00 / 2020-11-10 10:00:02\n"
" wal start/stop: 000000010000000000000001 / 000000010000000000000002\n"
" database size: 25.7MB, database backup size: 25.7MB\n"
" repo2: backup set size: 3MB, backup size: 3KB\n",
strZ(backupPath), strZ(backupPath), strZ(backupPath), strZ(backupPath), strZ(backupPath), strZ(backupPath),
strZ(backupPath), strZ(backupPath), strZ(backupPath)),
"text - multi-repo, multi-stanza cipher error");
// Backup label not found, one repo in error
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup label exists on one repo, other repo in error");
hrnCfgArgRawZ(argList2, cfgOptStanza, "stanza3");
hrnCfgArgRawZ(argList2, cfgOptSet, "20201110-100000F");
harnessCfgLoad(cfgCmdInfo, argList2);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza3\n"
" status: mixed\n"
" repo1: error (other)\n"
" [CryptoError] unable to load info file '%s/stanza3/backup.info' or '%s/stanza3/backup.info.copy':\n"
" CryptoError: cipher header invalid\n"
" HINT: is or was the repo encrypted?\n"
" FileMissingError: unable to open missing file '%s/stanza3/backup.info.copy' for read\n"
" HINT: backup.info cannot be opened and is required to perform a backup.\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use option --stanza if encryption settings are different for the stanza than the global"
" settings.\n"
" repo2: error (requested backup not found)\n"
" cipher: aes-256-cbc\n", strZ(backupPath), strZ(backupPath), strZ(backupPath)),
"backup label not found, one repo in error");
// Crypto error
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("WAL read error");
TEST_RESULT_VOID(
storagePathCreateP(
storageLocalWrite(), strNewFmt("%s/9.4-1", strZ(archiveStanza1Path)), .mode = 0200),
"WAL directory with bad permissions");
argList2 = strLstDup(argListMultiRepo);
hrnCfgArgRawZ(argList2, cfgOptStanza, "stanza1");
harnessCfgLoad(cfgCmdInfo, argList2);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza1\n"
" status: mixed\n"
" repo1: error (other)\n"
" [PathOpenError] unable to list file info for path '%s/stanza1/9.4-1': [13] Permission denied\n"
" repo2: error (no valid backups)\n"
" cipher: mixed\n"
" repo1: none\n"
" repo2: aes-256-cbc\n"
"\n"
" db (current)\n"
" wal archive min/max (9.5): 000000010000000000000003/000000010000000000000004\n", strZ(archivePath)),
"WAL directory read error");
// Unset environment key
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
@ -2591,21 +2752,82 @@ testRun(void)
TEST_ERROR_FMT(
harnessCfgLoad(cfgCmdInfo, argList), OptionInvalidError, "option 'set' not valid without option 'stanza'");
// Option --repo not required when only 1 repo configured
strLstAddZ(argList, "--stanza=stanza1");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("repo-level error");
TEST_RESULT_VOID(
storagePathCreateP(
storageLocalWrite(), strNewFmt("%s/repo2", testPath()), .mode = 0200), "repo directory with bad permissions");
argList = strLstNew();
hrnCfgArgKeyRawFmt(argList, cfgOptRepoPath, 1, "%s/repo2", testPath());
harnessCfgLoad(cfgCmdInfo, argList);
TEST_ERROR_FMT(
cmdInfo(), FileMissingError, "manifest does not exist for backup 'bogus'\n"
"HINT: is the backup listed when running the info command with --stanza option only?");
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: [invalid]\n"
" status: error (other)\n"
" [PathOpenError] unable to list file info for path '%s/repo2/backup': [13] Permission denied\n"
" cipher: none\n", testPath()),
"text - invalid stanza");
// Option --repo when only 1 repo configured but will search the repo provided
strLstAddZ(argList, "--repo=1");
hrnCfgArgRawZ(argList, cfgOptOutput, "json");
harnessCfgLoad(cfgCmdInfo, argList);
TEST_ERROR_FMT(
cmdInfo(), FileMissingError, "manifest does not exist for backup 'bogus'\n"
"HINT: is the backup listed when running the info command with --stanza option only?");
TEST_RESULT_STR(
infoRender(), strNewFmt(
"["
"{"
"\"archive\":[],"
"\"backup\":[],"
"\"cipher\":\"none\","
"\"db\":[],"
"\"name\":\"[invalid]\","
"\"repo\":["
"{"
"\"cipher\":\"none\","
"\"key\":1,"
"\"status\":{"
"\"code\":99,"
"\"message\":\"[PathOpenError] unable to list file info for path '%s/repo2/backup': [13] Permission"
" denied\""
"}"
"}"
"],"
"\"status\":{"
"\"code\":99,"
"\"lock\":{\"backup\":{\"held\":false}},"
"\"message\":\"other\""
"}"
"}"
"]", testPath()),
"json - invalid stanza");
argList = strLstNew();
hrnCfgArgKeyRawFmt(argList, cfgOptRepoPath, 1, "%s/repo2", testPath());
hrnCfgArgRawZ(argList, cfgOptStanza, "stanza1");
harnessCfgLoad(cfgCmdInfo, argList);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza1\n"
" status: error (other)\n"
" [PathOpenError] unable to list file info for path '%s/repo2/backup': [13] Permission denied\n"
" cipher: none\n", testPath()),
"text - stanza requested");
hrnCfgArgKeyRawFmt(argList, cfgOptRepoPath, 2, "%s/repo", testPath());
harnessCfgLoad(cfgCmdInfo, argList);
TEST_RESULT_STR(
infoRender(), strNewFmt(
"stanza: stanza1\n"
" status: mixed\n"
" repo1: error (other)\n"
" [PathOpenError] unable to list file info for path '%s/repo2/backup': [13] Permission denied\n"
" repo2: error (missing stanza path)\n"
" cipher: none\n", testPath()),
"text - stanza repo structure exists");
}
FUNCTION_HARNESS_RETURN_VOID();