db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2026-05-22 10:15:16 +02:00
Code Issues Releases Activity

Integration test improvements for disk and memory efficiency.

Browse Source 
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
This commit is contained in:
David Steele
2019-12-17 15:23:07 -05:00
parent ccea30b8d8
commit 977ec2e307
23 changed files with 533 additions and 415 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/expect/mock-expire-001.log
+12 -12
View File
@@ -299,7 +299,7 @@ backrest-checksum="[CHECKSUM]"
Nothing to expire
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=full --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 DETAIL: archive retention on backup [BACKUP-FULL-1], archiveId = 9.2-1, start = 000000010000000000000000
P00 DETAIL: no archive to remove, archiveId = 9.2-1
P00 INFO: expire command end: completed successfully
@@ -902,7 +902,7 @@ backrest-checksum="[CHECKSUM]"
Expire oldest full backup, archive expire falls on segment major boundary
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=full --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire full backup set: [BACKUP-FULL-1], [BACKUP-INCR-1]
P00 INFO: remove expired backup [BACKUP-INCR-1]
P00 INFO: remove expired backup [BACKUP-FULL-1]
@@ -1268,7 +1268,7 @@ backrest-checksum="[CHECKSUM]"
Expire oldest full backup
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=full --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire diff backup [BACKUP-DIFF-1]
P00 INFO: remove expired backup [BACKUP-DIFF-1]
P00 DETAIL: archive retention on backup [BACKUP-FULL-2], archiveId = 9.2-1, start = 000000010000000100000000
@@ -1921,7 +1921,7 @@ backrest-checksum="[CHECKSUM]"
Expire oldest diff backup, archive expire does not fall on major segment boundary
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire full backup set: [BACKUP-FULL-2], [BACKUP-DIFF-2]
P00 INFO: remove expired backup [BACKUP-DIFF-2]
P00 INFO: remove expired backup [BACKUP-FULL-2]
@@ -2043,7 +2043,7 @@ backrest-checksum="[CHECKSUM]"
Expire oldest diff backup (cascade to incr)
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire diff backup set: [BACKUP-DIFF-3], [BACKUP-INCR-2]
P00 INFO: remove expired backup [BACKUP-INCR-2]
P00 INFO: remove expired backup [BACKUP-DIFF-3]
@@ -2156,7 +2156,7 @@ backrest-checksum="[CHECKSUM]"
Expire archive based on newest incr backup
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=incr --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=incr --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=incr --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 DETAIL: archive retention on backup [BACKUP-FULL-3], archiveId = 9.2-1, start = 000000010000000200000010, stop = 000000010000000200000012
P00 DETAIL: archive retention on backup [BACKUP-DIFF-4], archiveId = 9.2-1, start = 00000001000000020000001E, stop = 000000010000000200000020
P00 DETAIL: archive retention on backup [BACKUP-INCR-3], archiveId = 9.2-1, start = 000000010000000200000024
@@ -2295,7 +2295,7 @@ backrest-checksum="[CHECKSUM]"
Expire diff treating full as diff
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=2 --repo1-retention-diff=1 --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=2 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --repo1-retention-diff=1 --repo1-retention-full=2 --stanza=db
P00 INFO: expire full backup set: [BACKUP-FULL-3], [BACKUP-DIFF-4], [BACKUP-INCR-3]
P00 INFO: expire diff backup [BACKUP-DIFF-5]
P00 INFO: remove expired backup [BACKUP-DIFF-5]
@@ -2430,7 +2430,7 @@ backrest-checksum="[CHECKSUM]"
Expire diff with repo-retention-archive with warning repo-retention-diff not set
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=diff --stanza=db
P00 WARN: option repo1-retention-full is not set, the repository may run out of space
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
P00 WARN: option 'repo1-retention-diff' is not set for 'repo1-retention-archive-type=diff'
@@ -2588,7 +2588,7 @@ backrest-checksum="[CHECKSUM]"
Expire full with repo-retention-archive with warning repo-retention-full not set
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-archive-type=full --repo1-retention-archive=1 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=1 --repo1-retention-archive-type=full --stanza=db
P00 WARN: option repo1-retention-full is not set, the repository may run out of space
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
P00 DETAIL: archive retention on backup [BACKUP-FULL-4], archiveId = 9.2-1, start = 00000001000000020000002A, stop = 00000001000000020000002C
@@ -2756,7 +2756,7 @@ backrest-checksum="[CHECKSUM]"
Expire no archive with warning since repo-retention-archive not set for INCR
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=1 --repo1-retention-diff=1 --repo1-retention-archive-type=incr expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive-type=incr --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive-type=incr --repo1-retention-diff=1 --repo1-retention-full=1 --stanza=db
P00 WARN: WAL segments will not be expired: option 'repo1-retention-archive-type=incr' but option 'repo1-retention-archive' is not set
P00 INFO: expire full backup [BACKUP-FULL-4]
P00 INFO: expire full backup [BACKUP-FULL-5]
@@ -2935,7 +2935,7 @@ backrest-checksum="[CHECKSUM]"
Expire no archive with warning since neither repo-retention-archive nor repo-retention-diff is set
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-archive-type=diff expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive-type=diff --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive-type=diff --stanza=db
P00 WARN: option repo1-retention-full is not set, the repository may run out of space
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
P00 WARN: WAL segments will not be expired: option 'repo1-retention-archive-type=diff' but neither option 'repo1-retention-archive' nor option 'repo1-retention-diff' is set
@@ -3125,7 +3125,7 @@ backrest-checksum="[CHECKSUM]"
Use oldest full backup for archive retention
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config="[TEST_PATH]/db-master/pgbackrest.conf" --stanza=db --log-level-console=detail --repo1-retention-full=10 --repo1-retention-diff=10 --repo1-retention-archive-type=full --repo1-retention-archive=10 expire
------------------------------------------------------------------------------------------------------------------------------------
P00 INFO: expire command begin [BACKREST-VERSION]: --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=trace --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log --log-subprocess --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=10 --repo1-retention-archive-type=full --repo1-retention-diff=10 --repo1-retention-full=10 --stanza=db
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-master/pgbackrest.conf --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --repo1-path=[TEST_PATH]/db-master/repo --repo1-retention-archive=10 --repo1-retention-archive-type=full --repo1-retention-diff=10 --repo1-retention-full=10 --stanza=db
P00 INFO: full backup total < 10 - using oldest full backup for 9.2-1 archive retention
P00 DETAIL: archive retention on backup [BACKUP-FULL-8], archiveId = 9.2-1, start = 000000010000000200000054
P00 DETAIL: remove archive: archiveId = 9.2-1, start = 00000001000000020000002A, stop = 000000010000000200000050