db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Code Issues Releases Activity
b489707793
pgbackrest/test/expect/real-all-001.log

892 lines
33 KiB
Plaintext
Raw Normal View History

Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
run 001 - bkp 1, sby 1, tls 0, dst backup, cmp none, storage posix, enc 0
=========================================================================
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Allow stanza-* commands to be run remotely. The stanza-create, stanza-upgrade and stanza-delete were required to be run on the repository host. When there was only one repository allowed this was not a problem. However, with the introduction of multiple repository support, this becomes more of a burden to the user, therefore the stanza-create, stanza-upgrade and stanza-delete commands have been improved to allow for them to be run remotely.
2021-03-10 15:10:46 +02:00
stanza-create db - main create stanza info files (db-primary host)
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --stanza=db stanza-create
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove real/all integration tests now covered by unit tests. Remove all check and stanza-* tests except for the ones that are intended to succeed. The successful tests show that the queries run with expected results against each version of PG which should also validate queries for the failure tests in the unit tests. Also remove the tests for --no-online backups since they don't require a database and are well tested in the unit tests.
2020-07-16 19:57:14 +02:00
check db - verify check command runs successfully (db-primary host)
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --archive-timeout=5 --stanza=db check
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove real/all integration tests now covered by unit tests. Remove all check and stanza-* tests except for the ones that are intended to succeed. The successful tests show that the queries run with expected results against each version of PG which should also validate queries for the failure tests in the unit tests. Also remove the tests for --no-online backups since they don't require a database and are well tested in the unit tests.
2020-07-16 19:57:14 +02:00
check db - verify check command runs successfully (backup host)
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --archive-timeout=5 --reset-pg2-host --reset-pg2-host-type --reset-pg2-host-cmd --reset-pg2-host-config --reset-pg2-host-user --reset-pg2-path --stanza=db check
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
full backup - fail on backup lock exists (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --type=full --stanza=db backup
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 22:21:50 +02:00
full backup - repo1 (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --buffer-size=[BUFFER-SIZE] --type=full --stanza=db backup
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[db]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-path=[TEST_PATH]/db-primary/db/base
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-port=6543
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-socket-path=[TEST_PATH]/db-primary/db
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
lock-path=[TEST_PATH]/db-primary/lock
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
log-path=[TEST_PATH]/db-primary/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
spool-path=[TEST_PATH]/db-primary/spool
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-path=[TEST_PATH]/backup/repo2
[global:backup]
archive-copy=y
start-fast=y
full backup - repo2 (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo=2 --type=full --stanza=db backup
------------------------------------------------------------------------------------------------------------------------------------
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg1-socket-path=[TEST_PATH]/db-primary/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
job-retry=0
lock-path=[TEST_PATH]/db-primary/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-primary/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
spool-path=[TEST_PATH]/db-primary/spool
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
job-retry=0
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
job-retry=0
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
repo2-path=[TEST_PATH]/backup/repo2
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[global:backup]
archive-copy=y
start-fast=y
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
full backup - with disabled expire-auto (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=1 --no-expire-auto --type=full --stanza=db backup
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
------------------------------------------------------------------------------------------------------------------------------------
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg1-socket-path=[TEST_PATH]/db-primary/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
lock-path=[TEST_PATH]/db-primary/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-primary/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
spool-path=[TEST_PATH]/db-primary/spool
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-path=[TEST_PATH]/backup/repo2
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
[global:backup]
archive-copy=y
start-fast=y
expire full=1 (backup host)
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 22:21:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=1 --repo=1 --stanza=db expire
Add expire-auto option. This allows automatic expiration after a successful backup to be disabled.
2020-07-14 14:12:25 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
restore, type 'standby', remap - restore backup on replica (db-standby host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-standby [BACKREST-BIN] --config=[TEST_PATH]/db-standby/pgbackrest.conf --recovery-option="primary_conninfo=host=db-primary port=6543 user=replicator" --type=standby --link-map="pg_xlog=[TEST_PATH]/db-standby/db/pg_xlog" --link-all --stanza=db restore
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
+ supplemental file: [TEST_PATH]/db-standby/db/base/recovery.conf
-----------------------------------------------------------------
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
primary_conninfo = 'host=db-primary port=6543 user=replicator'
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-standby/pgbackrest.conf --stanza=db archive-get %f "%p"'
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
standby_mode = 'on'
full backup - backup from standby, failure to access at least one standby (backup host)
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --pg256-host=bogus --backup-standby --type=full --stanza=db backup
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
full backup - backup from standby (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=1 --backup-standby --type=full --stanza=db backup
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
[db]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-path=[TEST_PATH]/db-primary/db/base
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
pg1-port=6543
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-socket-path=[TEST_PATH]/db-primary/db
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/db-primary/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-primary/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
spool-path=[TEST_PATH]/db-primary/spool
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[db:restore]
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
[global]
archive-async=y
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
compress-level=3
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-path=[TEST_PATH]/backup/repo2
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
[global:backup]
archive-copy=y
start-fast=y
check db - verify check command on standby (db-standby host)
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
> [CONTAINER-EXEC] db-standby [BACKREST-BIN] --config=[TEST_PATH]/db-standby/pgbackrest.conf --reset-pg2-host --reset-pg2-host-type --reset-pg2-host-cmd --reset-pg2-host-config --reset-pg2-host-user --reset-pg2-path --stanza=db check
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
diff backup - backup for adhoc expire (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --type=diff --stanza=db backup
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg1-socket-path=[TEST_PATH]/db-primary/db
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
lock-path=[TEST_PATH]/db-primary/lock
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
log-path=[TEST_PATH]/db-primary/log
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
spool-path=[TEST_PATH]/db-primary/spool
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[db:restore]
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
[global]
archive-async=y
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-path=[TEST_PATH]/backup/repo2
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
[global:backup]
archive-copy=y
start-fast=y
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
stop all stanzas (db-primary host)
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf stop
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
incr backup - attempt backup when stopped (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db backup
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
start all stanzas (db-primary host)
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf start
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
incr backup - fail on archive_mode=always (backup host)
Make --repo optional for backup command. If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 21:49:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db backup
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
expire --set=[BACKUP-DIFF-1] (backup host)
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 22:21:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --set=[BACKUP-DIFF-1] --repo=1 --stanza=db expire
Add --set option to the expire command. The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 20:00:36 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 22:21:50 +02:00
incr backup - delta (backup host)
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stop-auto --buffer-size=[BUFFER-SIZE] --delta --repo=2 --stanza=db backup
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/pgbackrest.conf
-----------------------------------------------------------
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[db]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-path=[TEST_PATH]/db-primary/db/base
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-port=6543
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
pg1-socket-path=[TEST_PATH]/db-primary/db
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
lock-path=[TEST_PATH]/db-primary/lock
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
log-path=[TEST_PATH]/db-primary/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
spool-path=[TEST_PATH]/db-primary/spool
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
+ supplemental file: [TEST_PATH]/db-standby/pgbackrest.conf
-----------------------------------------------------------
[db]
pg1-path=[TEST_PATH]/db-standby/db/base
pg1-port=6544
pg1-socket-path=[TEST_PATH]/db-standby/db
[db:restore]
[global]
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/db-standby/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/db-standby/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo1-host-user=[USER-1]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-host=backup
repo2-host-cmd=[BACKREST-BIN]
repo2-host-config=[TEST_PATH]/backup/pgbackrest.conf
repo2-host-user=[USER-1]
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
spool-path=[TEST_PATH]/db-standby/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
pg1-host=db-primary
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-primary/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-primary/db/base
pg1-port=6543
pg2-host=bogus
pg2-host-cmd=[BACKREST-BIN]
pg2-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg2-host-user=[USER-1]
pg2-path=[TEST_PATH]/db-standby/db/base
Increase max index allowed for pg/repo options to 256. The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained. Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly. Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX. Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior. cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 18:09:50 +02:00
pg256-host=db-standby
pg256-host-cmd=[BACKREST-BIN]
pg256-host-config=[TEST_PATH]/db-standby/pgbackrest.conf
pg256-host-user=[USER-1]
pg256-path=[TEST_PATH]/db-standby/db/base
pg256-port=6544
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
[global]
archive-async=y
buffer-size=[BUFFER-SIZE]
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 21:24:14 +02:00
bundle=y
bundle-size=1MiB
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
compress-level=3
compress-level-network=1
compress-type=none
db-timeout=45
Add job-retry and job-retry-interval options. These options specify the number of local worker job retries and the retry interval after one immediate retry. There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests. Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally. These options are currently internal.
2021-01-11 22:15:25 +02:00
job-retry=0
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
lock-path=[TEST_PATH]/backup/lock
log-level-console=detail
log-level-file=[LOG-LEVEL-FILE]
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
log-subprocess=[LOG-SUBPROCESS]
log-timestamp=n
protocol-timeout=60
repo1-path=[TEST_PATH]/backup/repo
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
repo2-path=[TEST_PATH]/backup/repo2
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
[global:backup]
archive-copy=y
start-fast=y
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
check db - check command with tablespace (backup host)
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --archive-timeout=5 --reset-pg2-host --reset-pg2-host-type --reset-pg2-host-cmd --reset-pg2-host-config --reset-pg2-host-user --reset-pg2-path --stanza=db check
Fix manifest test in the check command when tablespaces are present. Fixed by Cynthia Shang. Reported by Thomas Flatley.
2018-03-20 01:54:03 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Use PostgreSQL instead of postmaster where appropriate. Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.
2020-06-17 21:14:59 +02:00
restore, type 'default', expect exit 38 - pg running (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --link-all --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore, type 'default', expect exit 40 - path not empty (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --link-all --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore, type 'default' (db-primary host)
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --db-include=test2 --db-include=test3 --buffer-size=[BUFFER-SIZE] --link-all --repo=2 --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
restore_command = '[BACKREST-BIN] --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/db-primary/pgbackrest.conf --repo=2 --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore, force, backup '[BACKUP-FULL-1]', type 'immediate', target-action=promote (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --force --set=[BACKUP-FULL-1] --type=immediate --link-all --target-action=promote --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target = 'immediate'
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
recovery_target_action = 'promote'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore, force, backup '[BACKUP-INCR-1]', type 'xid', target '[XID-TARGET-1]', target-action=promote (db-primary host)
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --force --set=[BACKUP-INCR-1] --tablespace-map-all=../../tablespace --type=xid --target="[XID-TARGET-1]" --link-all --target-action=promote --repo=2 --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --repo=2 --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_xid = '[XID-TARGET-1]'
recovery_target_action = 'promote'
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore, type 'preserve' (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --type=preserve --link-all --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --repo=2 --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_xid = '[XID-TARGET-1]'
recovery_target_action = 'promote'
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore delta, type 'time', target '[TIMESTAMP-TARGET-1]', target-action=promote (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --delta --type=time --target="[TIMESTAMP-TARGET-1]" --link-all --target-action=promote --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_time = '[TIMESTAMP-TARGET-1]'
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
recovery_target_action = 'promote'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore delta, backup '[BACKUP-INCR-1]', type 'xid', target '[XID-TARGET-1]', exclusive, target-action=promote (db-primary host)
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --delta --set=[BACKUP-INCR-1] --type=xid --target="[XID-TARGET-1]" --target-exclusive --link-all --target-action=promote --repo=2 --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --repo=2 --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_xid = '[XID-TARGET-1]'
recovery_target_inclusive = 'false'
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
recovery_target_action = 'promote'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore delta, force, type 'name', target 'backrest', target-action=promote (db-primary host)
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --delta --force --type=name --target="backrest" --link-all --target-action=promote --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Remove restore default repo from integration tests. The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 18:32:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --stanza=db archive-get %f "%p"'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_name = 'backrest'
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
recovery_target_action = 'promote'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
restore delta, backup '[BACKUP-INCR-1]', type 'standby', timeline '4' (db-primary host)
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --delta --set=[BACKUP-INCR-1] --type=standby --target-timeline="4" --link-all --repo=2 --stanza=db restore
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
+ supplemental file: [TEST_PATH]/db-primary/db/base/recovery.conf
-----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Multiple repository support. Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy. Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified. Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later. The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient. For single repository configurations there should be no change in behavior.
2021-03-08 20:31:13 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf --repo=2 --stanza=db archive-get %f "%p"'
Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf. This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal. The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 23:39:45 +02:00
standby_mode = 'on'
More flexible configuration for databases Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 18:39:27 +02:00
recovery_target_timeline = '4'
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
stop all stanzas (db-primary host)
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf stop
------------------------------------------------------------------------------------------------------------------------------------
stop db stanza (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db stop
------------------------------------------------------------------------------------------------------------------------------------
stanza-delete db - delete stanza with --force when pgbackrest on pg host not accessible (backup host)
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 22:21:50 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo=1 --stanza=db --force stanza-delete
Simplify test matrix for real/all tests. Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added. Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro. Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it. Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6. The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them. The following is an example of the new test log (note longer runtime of each test): module=real, test=all, run=1, pg-version=10 (106.91s) module=real, test=all, run=1, pg-version=9.5 (151.09s) module=real, test=all, run=1, pg-version=9.2 (123.11s) module=real, test=all, run=1, pg-version=9.1 (129s) vs. the old test log (sub-second tests were skipped entirely): module=real, test=all, run=2, pg-version=10 (0.31s) module=real, test=all, run=3, pg-version=10 (0.26s) module=real, test=all, run=4, pg-version=10 (60.39s) module=real, test=all, run=1, pg-version=10 (69.12s) module=real, test=all, run=6, pg-version=10 (34s) module=real, test=all, run=5, pg-version=10 (42.75s) module=real, test=all, run=2, pg-version=9.5 (0.21s) module=real, test=all, run=3, pg-version=9.5 (0.21s) module=real, test=all, run=4, pg-version=9.5 (0.21s) module=real, test=all, run=5, pg-version=9.5 (0.26s) module=real, test=all, run=6, pg-version=9.5 (0.21s) module=real, test=all, run=1, pg-version=9.2 (72.78s) module=real, test=all, run=2, pg-version=9.2 (0.26s) module=real, test=all, run=3, pg-version=9.2 (0.31s) module=real, test=all, run=4, pg-version=9.2 (0.21s) module=real, test=all, run=5, pg-version=9.2 (0.21s) module=real, test=all, run=6, pg-version=9.2 (0.21s) module=real, test=all, run=1, pg-version=9.5 (88.41s) module=real, test=all, run=2, pg-version=9.1 (0.21s) module=real, test=all, run=3, pg-version=9.1 (0.26s) module=real, test=all, run=4, pg-version=9.1 (0.21s) module=real, test=all, run=5, pg-version=9.1 (0.31s) module=real, test=all, run=6, pg-version=9.1 (0.26s) module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 19:44:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
start all stanzas (db-primary host)
> [CONTAINER-EXEC] db-primary [BACKREST-BIN] --config=[TEST_PATH]/db-primary/pgbackrest.conf start
------------------------------------------------------------------------------------------------------------------------------------
start all stanzas (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf start
------------------------------------------------------------------------------------------------------------------------------------
Reference in New Issue Copy Permalink