db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Code Issues Releases Activity
b5dd14e6f3
pgbackrest/test/expect/mock-all-002.log

2636 lines
134 KiB
Plaintext
Raw Normal View History

Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
run 002 - rmt 1, storage s3, enc 1, delta 0
===========================================
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
stanza-create db - create required data for stanza (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-create
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
[db]
db-catalog-version=201409291
db-control-version=942
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
db-version="9.4"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
[db]
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
db-version="9.4"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000094,"db-version":"9.4"}
Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. Contributed by Cynthia Shang.
2016-12-20 23:52:20 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
full backup - create pg_stat link, pg_clog dir (backup host)
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --manifest-save-threshold=3 --cmd-ssh=/usr/bin/ssh --pg1-port=9999 --pg1-socket-path=/test_socket_path --buffer-size=[BUFFER-SIZE] --checksum-page --process-max=1 --type=full --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: invalid page checksums found in file db-master:[TEST_PATH]/db-master/db/base/base/32768/33001 at pages 0, 3-5, 7
P00 WARN: invalid page checksum found in file db-master:[TEST_PATH]/db-master/db/base/base/16384/17000 at page 1
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-FULL-1]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-FULL-1]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="full"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base","type":"path"}
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
pg_data/pg_hba.conf={"file":"pg_hba.conf","path":"../pg_config","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={"path":"../pg_stat","type":"link"}
pg_data/postgresql.conf={"file":"postgresql.conf","path":"../pg_config","type":"link"}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_hba.conf={"checksum":"dd4cea0cae348309f9de28ad4ded8ee2cc2e6d5b","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Fixed a issue where tablespaces were copied from the master during standby backup.
2016-09-04 15:19:44 +02:00
master=false
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
pg_data/pg_hba.conf={"destination":"../pg_config/pg_hba.conf"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={"destination":"../pg_stat"}
pg_data/postgresql.conf={"destination":"../pg_config/postgresql.conf"}
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
pg_data/pg_stat={}
pg_data/pg_tblspc={}
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
full backup - resume (backup host)
Remove obsolete integration tests from mock/all. The protocol timeout tests have been superceded by unit tests. The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto. It didn't hurt anything but it did add 200ms to each test where it appeared. Catalog and control version tests were redundant. The database version and system id tests covered the important code paths and the C code gets these values from a lookup table. Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 23:06:27 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --force --checksum-page --type=full --stanza=db backup
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Update missing manifest warning in BackupInfo. This brings the Perl message in line with C to reduce expect log churn.
2019-11-25 15:51:28 +02:00
P00 WARN: backup '[BACKUP-FULL-1]' missing manifest removed from backup.info
Show Process ID in log output instead of filtering it out with the timestamp.
2016-09-14 23:30:57 +02:00
P00 WARN: --no-online passed and postmaster.pid exists but --force was passed so backup will continue though it looks like the postmaster is running and the backup will probably not be consistent
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: resumable backup [BACKUP-FULL-2] of same type exists -- remove invalid files and resume
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
Fix exclusions for special files. Prior to 2.16 the Perl manifest code would skip any file that began with a dot. This was not intentional but it allowed PostgreSQL socket files to be located in the data directory. The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors. Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory. Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion. The warning can be suppressed with an explicit --exclude. Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 13:47:54 +02:00
exclude=postgresql.auto.conf
exclude=pg_log/
exclude=pg_log2
exclude=apipe
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-FULL-2]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="full"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base","type":"path"}
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
pg_data/pg_hba.conf={"file":"pg_hba.conf","path":"../pg_config","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={"path":"../pg_stat","type":"link"}
pg_data/postgresql.conf={"file":"postgresql.conf","path":"../pg_config","type":"link"}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changecontent.txt={"checksum":"238a131a3e8eb98d1fc5b27d882ca40b7618fd2a","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_hba.conf={"checksum":"dd4cea0cae348309f9de28ad4ded8ee2cc2e6d5b","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
pg_data/pg_hba.conf={"destination":"../pg_config/pg_hba.conf"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={"destination":"../pg_stat"}
pg_data/postgresql.conf={"destination":"../pg_config/postgresql.conf"}
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Major refactor of the test suite. * Make the code more modular and object-oriented. * Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 14:12:58 +02:00
restore delta, backup '[BACKUP-FULL-2]' - add and delete files (db-master host)
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --set=[BACKUP-FULL-2] --link-all --cmd-ssh=/usr/bin/ssh --stanza=db restore
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base/recovery.conf
----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
restore_command = '[BACKREST-BIN] --cmd-ssh=/usr/bin/ssh --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Added vagrant configuration for Ubuntu 12.04. Changes to regression tests to work better with VMs.
2015-07-07 03:00:48 +02:00
User/group permissions improvements. Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
2016-06-27 03:01:20 +02:00
restore delta, backup '[BACKUP-FULL-2]' - fix broken symlink (db-master host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --set=[BACKUP-FULL-2] --link-all --compress-level-network=0 --stanza=db restore
User/group permissions improvements. Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
2016-06-27 03:01:20 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Allow any non-boolean, non-command-line option to be negated. This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs. Note that boolean, non-command-line options are already negatable.
2017-12-05 22:23:32 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base/recovery.conf
----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --compress-level-network=0 --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Fixed an issue that suppressed locality errors for backup and restore. When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host. Reported by Lardière Sébastien.
2017-12-05 21:59:09 +02:00
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
restore delta, force, backup '[BACKUP-FULL-2]' - restore links as directories (db-master host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --force --set=[BACKUP-FULL-2] --stanza=db restore
Closed #127: More sanity checking for --delta restores Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-06 06:03:29 +02:00
------------------------------------------------------------------------------------------------------------------------------------
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 13:52:02 +02:00
P00 WARN: file link 'pg_hba.conf' will be restored as a file at the same location
P00 WARN: contents of directory link 'pg_stat' will be restored in a directory at the same location
P00 WARN: file link 'postgresql.conf' will be restored as a file at the same location
P00 WARN: unknown user 'bogus' in backup manifest mapped to current user
P00 WARN: unknown group 'bogus' in backup manifest mapped to current group
Closed #127: More sanity checking for --delta restores Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-06 06:03:29 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base/recovery.conf
----------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Closed #127: More sanity checking for --delta restores Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-06 06:03:29 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
incr backup - add tablespace 1 (backup host)
Remove test point infrastructure. 82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code. Since tests points are no longer used remove the infrastructure. Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 20:16:47 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
P00 WARN: incr backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-FULL-2]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base/pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt: file size 7 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-INCR-1]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-INCR-1]"
backup-prior="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="incr"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base","type":"path"}
pg_tblspc/1={"path":"[TEST_PATH]/db-master/db/tablespace/ts1","tablespace-id":"1","tablespace-name":"ts1","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changecontent.txt={"checksum":"238a131a3e8eb98d1fc5b27d882ca40b7618fd2a","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt={"checksum":"14c44cef6287269b08d41de489fd492bb9fc795d","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/1={"destination":"[TEST_PATH]/db-master/db/tablespace/ts1"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/1={}
pg_tblspc/1/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/1/[TS_PATH-1]/16384={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
incr backup - resume and add tablespace 2 (backup host)
Remove obsolete integration tests from mock/all. The protocol timeout tests have been superceded by unit tests. The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto. It didn't hurt anything but it did add 200ms to each test where it appeared. Catalog and control version tests were redundant. The database version and system id tests covered the important code paths and the C code gets these values from a lookup table. Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 23:06:27 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --process-max=1 --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Update missing manifest warning in BackupInfo. This brings the Perl message in line with C to reduce expect log churn.
2019-11-25 15:51:28 +02:00
P00 WARN: backup '[BACKUP-INCR-1]' missing manifest removed from backup.info
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
P00 WARN: incr backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-FULL-2]
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: file 'changetime.txt' has timestamp earlier than prior backup, enabling delta checksum
P00 WARN: resumable backup [BACKUP-INCR-2] of same type exists -- remove invalid files and resume
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base/pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt: file size 7 is not divisible by page size 8192
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt: file size 7 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-INCR-2]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-INCR-2]"
backup-prior="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="incr"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
option-delta=true
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base","type":"path"}
pg_tblspc/1={"path":"[TEST_PATH]/db-master/db/tablespace/ts1","tablespace-id":"1","tablespace-name":"ts1","type":"link"}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_tblspc/11={"path":"[TEST_PATH]/db-master/db/tablespace/ts11","tablespace-id":"11","tablespace-name":"ts11","type":"link"}
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt={"checksum":"d85de07d6421d90aa9191c11c889bfde43680f0f","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/1={"destination":"[TEST_PATH]/db-master/db/tablespace/ts1"}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/pg_tblspc/11={"destination":"[TEST_PATH]/db-master/db/tablespace/ts11"}
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/1={}
pg_tblspc/1/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/1/[TS_PATH-1]/16384={}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_tblspc/11={}
pg_tblspc/11/[TS_PATH-1]={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
diff backup - drop tablespace 11 (backup host)
Remove obsolete integration tests from mock/all. The protocol timeout tests have been superceded by unit tests. The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto. It didn't hurt anything but it did add 200ms to each test where it appeared. Catalog and control version tests were redundant. The database version and system id tests covered the important code paths and the C code gets these values from a lookup table. Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 23:06:27 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --process-max=1 --type=diff --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
P00 WARN: diff backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-FULL-2]
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: file 'changetime.txt' has timestamp earlier than prior backup, enabling delta checksum
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base/pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt: file size 7 is not divisible by page size 8192
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt: file size 7 is not divisible by page size 8192
Added resume setting to allow disabling resume feature.
2015-05-07 18:29:30 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-DIFF-1]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-DIFF-1]"
backup-prior="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="diff"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
option-delta=true
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base","type":"path"}
pg_tblspc/1={"path":"[TEST_PATH]/db-master/db/tablespace/ts1","tablespace-id":"1","tablespace-name":"ts1","type":"link"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/1/[TS_PATH-1]/16384/tablespace1.txt={"checksum":"d85de07d6421d90aa9191c11c889bfde43680f0f","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/1={"destination":"[TEST_PATH]/db-master/db/tablespace/ts1"}
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/1={}
pg_tblspc/1/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/1/[TS_PATH-1]/16384={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
restore, backup '[BACKUP-DIFF-1]', remap - remap all paths (db-master host)
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --set=[BACKUP-DIFF-1] --stanza=db restore
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base-2/recovery.conf
------------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Added vagrant configuration for Ubuntu 12.04. Changes to regression tests to work better with VMs.
2015-07-07 03:00:48 +02:00
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
restore delta, backup '[BACKUP-DIFF-1]', remap - ensure file in tblspc root remains after --delta (db-master host)
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --set=[BACKUP-DIFF-1] --stanza=db restore
Closed #58: Get catalog number for better tablespace copying When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 21:33:15 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base-2/recovery.conf
------------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Closed #58: Get catalog number for better tablespace copying When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 21:33:15 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
incr backup - add files and remove tablespace 2 (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --process-max=1 --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
P00 WARN: incr backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-DIFF-1]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2b.txt: file size 8 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[db:restore]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-INCR-3]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-INCR-3]"
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
backup-prior="[BACKUP-DIFF-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="incr"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"e0101dd8ffb910c9c202ca35b5f828bcb9697bed","checksum-page":false,"checksum-page-error":[1],"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Update Perl page checksum expression. This expression determines which files contain page checksums but it was also including the directory above the relation directories. In a real PostgreSQL installation this not a problem because these directories don't contain any files. However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no. Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 14:37:09 +02:00
pg_data/base/base2.txt={"checksum":"09b5e31766be1dba1ec27de82f975c1b6eea2a92","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/zerosize.txt={"master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2b.txt={"checksum":"e324463005236d83e6e54795dbddd20a74533bf3","checksum-page":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Fixed a issue where tablespaces were copied from the master during standby backup.
2016-09-04 15:19:44 +02:00
master=false
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-DIFF-1]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
incr backup - update files (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
P00 WARN: incr backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-INCR-3]
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: file 'base/16384/17000' has same timestamp as prior but different size, enabling delta checksum
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/base/16384/17000: file size 8 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[db:restore]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-INCR-4]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-INCR-4]"
backup-prior="[BACKUP-INCR-3]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="incr"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
option-delta=true
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"9a53d532e27785e681766c98516a5e93f096a501","checksum-page":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Update Perl page checksum expression. This expression determines which files contain page checksums but it was also including the directory above the relation directories. In a real PostgreSQL installation this not a problem because these directories don't contain any files. However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no. Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 14:37:09 +02:00
pg_data/base/base2.txt={"checksum":"09b5e31766be1dba1ec27de82f975c1b6eea2a92","reference":"[BACKUP-INCR-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
pg_data/zerosize.txt={"master":true,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"reference":"[BACKUP-DIFF-1]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2b.txt={"checksum":"e324463005236d83e6e54795dbddd20a74533bf3","checksum-page":false,"reference":"[BACKUP-INCR-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Fixed a issue where tablespaces were copied from the master during standby backup.
2016-09-04 15:19:44 +02:00
master=false
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-DIFF-1]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-INCR-3]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]","[BACKUP-INCR-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
diff backup - updates since last full (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --process-max=1 --type=diff --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
P00 WARN: diff backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-FULL-2]
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: file 'base/16384/17000' has same timestamp as prior but different size, enabling delta checksum
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/base/16384/17000: file size 8 is not divisible by page size 8192
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2b.txt: file size 8 is not divisible by page size 8192
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt: file size 7 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[db:restore]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
+ supplemental file: /backup/db/[BACKUP-DIFF-2]/backup.manifest
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
backup-label="[BACKUP-DIFF-2]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backup-prior="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="diff"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
option-delta=true
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"9a53d532e27785e681766c98516a5e93f096a501","checksum-page":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Update Perl page checksum expression. This expression determines which files contain page checksums but it was also including the directory above the relation directories. In a real PostgreSQL installation this not a problem because these directories don't contain any files. However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no. Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 14:37:09 +02:00
pg_data/base/base2.txt={"checksum":"09b5e31766be1dba1ec27de82f975c1b6eea2a92","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"master":true,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"master":true,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2b.txt={"checksum":"e324463005236d83e6e54795dbddd20a74533bf3","checksum-page":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Fixed a issue where tablespaces were copied from the master during standby backup.
2016-09-04 15:19:44 +02:00
master=false
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-DIFF-1]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-INCR-3]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]","[BACKUP-INCR-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-DIFF-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
diff backup - remove files (backup host)
Update integration tests in mock/all that use test points. Test points will not be available in the C code so update these tests as best as possible without using them. This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.
2019-11-25 14:48:52 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --process-max=1 --type=diff --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
P00 WARN: diff backup cannot alter compress-type option to 'lz4', reset to value in [BACKUP-FULL-2]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
P00 WARN: diff backup cannot alter 'checksum-page' option to 'false', reset to 'true' from [BACKUP-FULL-2]
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: file 'changetime.txt' has timestamp earlier than prior backup, enabling delta checksum
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2c.txt: file size 12 is not divisible by page size 8192
P00 WARN: page misalignment in file db-master:[TEST_PATH]/db-master/db/base-2/pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt: file size 7 is not divisible by page size 8192
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[db:restore]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
compress-type=lz4
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
+ supplemental file: /backup/db/[BACKUP-DIFF-3]/backup.manifest
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
backup-label="[BACKUP-DIFF-3]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backup-prior="[BACKUP-FULL-2]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="diff"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=true
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-compress=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
option-compress-type="none"
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
option-delta=true
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=1
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-3]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","checksum-page":true,"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","checksum-page":false,"checksum-page-error":[0,[3,5],7],"master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
pg_data/zero_from_start={"reference":"[BACKUP-FULL-2]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2c.txt={"checksum":"dfcb8679956b734706cf87259d50c88f83e80e66","checksum-page":false,"master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-DIFF-1]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-INCR-3]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]","[BACKUP-INCR-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-DIFF-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
full backup - update file (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --type=full --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[db:restore]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
compress-type=lz4
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/[BACKUP-FULL-3]/backup.manifest
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
backup-label="[BACKUP-FULL-3]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="full"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-compress=true
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
option-compress-type="lz4"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-4]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"mode":"0660","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"7579ada0808d7f98087a0a586d0df9de009cdc33","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/zero_from_start={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/zerosize.txt={"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2c.txt={"checksum":"dfcb8679956b734706cf87259d50c88f83e80e66","master":false,"repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
master=true
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-INCR-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-DIFF-1]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-INCR-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-INCR-3]","backup-reference":["[BACKUP-FULL-2]","[BACKUP-DIFF-1]","[BACKUP-INCR-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"incr","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-DIFF-2]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-DIFF-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-2]","backup-reference":["[BACKUP-FULL-2]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":true,"option-compress":false,"option-hardlink":false,"option-online":false}
[BACKUP-FULL-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
expire full=1 (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=1 --stanza=db expire
Closed #131: Offline backups make expire command fail.
2016-01-16 06:37:17 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
diff backup - add file (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --checksum-page --type=diff --stanza=db backup
Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests.
2015-05-05 19:08:48 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
Added tablespace setting to allow compact database restores for development, staging, etc.
2015-05-09 00:34:27 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[db:restore]
tablespace-map=1=[TEST_PATH]/db-master/db/tablespace/ts1-2
tablespace-map=2=[TEST_PATH]/db-master/db/tablespace/ts2-2
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2
New simpler configuration and consistent project/exe/path naming. * The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required. * Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. * The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally. * The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. * Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory. * Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory. * Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 15:30:54 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
compress-type=lz4
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
db-timeout=45
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Added the log-level-stderr option. Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. Suggested by Sascha Biberhofer.
2016-10-05 15:09:30 +02:00
log-level-stderr=off
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[global:backup]
archive-copy=y
start-fast=y
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
+ supplemental file: /backup/db/[BACKUP-DIFF-4]/backup.manifest
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
---------------------------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup]
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
backup-label="[BACKUP-DIFF-4]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backup-prior="[BACKUP-FULL-3]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="diff"
[backup:db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
option-archive-copy=true
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 17:25:46 +02:00
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Added the --checksum-page option. This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-13 01:54:07 +02:00
option-checksum-page=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-compress=true
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
option-compress-type="lz4"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
option-hardlink=false
Closed #167: Rename `--no-start-stop` option to `--no-online`.
2016-02-20 02:32:02 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[backup:target]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data={"path":"[TEST_PATH]/db-master/db/base-2","type":"path"}
pg_tblspc/2={"path":"[TEST_PATH]/db-master/db/tablespace/ts2-2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-4]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"7579ada0808d7f98087a0a586d0df9de009cdc33","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/base2.txt={"checksum":"cafac3c59553f2cfde41ce2e62e7662295f108c0","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. Contributed by Cynthia Shang.
2018-08-24 22:50:33 +02:00
pg_data/zero_from_start={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add zero-length file to mock/all test.
2018-07-05 21:40:50 +02:00
pg_data/zerosize.txt={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2c.txt={"checksum":"dfcb8679956b734706cf87259d50c88f83e80e66","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:file:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Fixed a issue where tablespaces were copied from the master during standby backup.
2016-09-04 15:19:44 +02:00
master=false
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link]
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
pg_data/pg_tblspc/2={"destination":"[TEST_PATH]/db-master/db/tablespace/ts2-2"}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:link:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path]
pg_data={}
pg_data/base={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
pg_tblspc/2/[TS_PATH-1]/32768={}
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
[target:path:default]
Continuous integration using travis-ci.
2017-02-21 15:59:23 +02:00
group="[GROUP-1]"
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[backrest]
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
backrest-format=5
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-DIFF-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-3]","backup-reference":["[BACKUP-FULL-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db]
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-catalog-version=201409291
db-control-version=942
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Database version refactoring. * Refactor db version constants into a separate module. * Update synthetic backup tests to PostgreSQL 9.4.
2016-08-12 04:35:24 +02:00
db-version="9.4"
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Major refactor of the test suite. * Make the code more modular and object-oriented. * Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 14:12:58 +02:00
restore delta, remap - selective restore 16384 (db-master host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --db-include=16384 --stanza=db restore
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base-2/recovery.conf
------------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
Major refactor of the test suite. * Make the code more modular and object-oriented. * Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 14:12:58 +02:00
restore delta, remap - selective restore 32768 (db-master host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --db-include=32768 --stanza=db restore
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base-2/recovery.conf
------------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. Suggested by Yogesh Sharma.
2017-04-12 18:41:34 +02:00
restore delta, remap, expect exit 80 - error on invalid id (db-master host)
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --log-level-console=warn --db-include=7777 --stanza=db restore
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. Suggested by Yogesh Sharma.
2017-04-12 18:41:34 +02:00
P00 ERROR: [080]: database to include '7777' does not exist
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. Suggested by Yogesh Sharma.
2017-04-12 18:41:34 +02:00
restore delta, remap, expect exit 81 - error on system id (db-master host)
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --delta --log-level-console=warn --db-include=1 --stanza=db restore
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. Suggested by Yogesh Sharma.
2017-04-12 18:41:34 +02:00
P00 ERROR: [081]: system databases (template0, postgres, etc.) are included by default
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 15:21:39 +02:00
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 13:52:02 +02:00
restore, remap - no tablespace remap (db-master host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --tablespace-map-all=../../tablespace --stanza=db restore
New manifest format. * All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location. * Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function. * Added detail log level which will output more information than info without being as verbose as debug.
2016-04-15 04:50:02 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Rename test paths for clarity. This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved. Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 16:54:31 +02:00
+ supplemental file: [TEST_PATH]/db-master/db/base-2/base/recovery.conf
-----------------------------------------------------------------------
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
# Recovery settings generated by pgBackRest restore on [TIMESTAMP]
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
restore_command = '[BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get %f "%p"'
Implemented issue #26: Info command. * Includes updating the manifest to format 4. It turns out the manifest and .info files were not very good for providing information. A format update was required anyway so worked through the backlog of changes that would require a format change. * Multiple database versions are now supported in the archive. Does't actually work yet but the structure should be good. * Tests use more constants now that test logs can catch name regressions.
2015-06-14 00:25:49 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
diff backup - option backup-standby reset - backup performed from master (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --no-online --log-level-console=info --backup-standby --type=diff --stanza=db backup
------------------------------------------------------------------------------------------------------------------------------------
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
P00 INFO: backup command begin [BACKREST-VERSION]: --backup-standby --buffer-size=[BUFFER-SIZE] --compress-level=3 --compress-level-network=1 --compress-type=lz4 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=info --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base-2/base --process-max=2 --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db --start-fast --type=diff
Add time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 21:25:03 +02:00
P00 WARN: option 'repo1-retention-full' is not set for 'repo1-retention-full-type=count', the repository may run out of space
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-full' to the maximum.
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
P00 WARN: option backup-standby is enabled but backup is offline - backups will be performed from the primary
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 INFO: last backup label = [BACKUP-FULL-3], version = [VERSION-1]
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base-2/base/base/base2.txt (9B, 100%) checksum cafac3c59553f2cfde41ce2e62e7662295f108c0
P00 INFO: diff backup size = 9B
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
P00 INFO: new backup label = [BACKUP-DIFF-5]
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 INFO: http statistics:[HTTP-STATISTICS]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 INFO: backup command end: completed successfully
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/backup/pgbackrest.conf --lock-path=[TEST_PATH]/backup/lock --log-level-console=info --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Rename retention-* options to repo-retention-*.
2018-02-19 22:07:24 +02:00
P00 INFO: option 'repo1-retention-archive' is not set - archive logs will not be expired
The expire command is implemented entirely in C. This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests. Along the way at least one bug was fixed, see issue #748. Contributed by Cynthia Shang.
2019-06-18 21:19:20 +02:00
P00 INFO: http statistics:[HTTP-STATISTICS]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
P00 INFO: expire command end: completed successfully
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[db:restore]
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
compress-type=none
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
repo1-host-user=[USER-1]
Consolidate archive-push tests into new archive test.
2017-06-21 14:38:53 +02:00
spool-path=[TEST_PATH]/db-master/spool
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
pg1-host-user=[USER-1]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base-2/base
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
compress-type=lz4
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
log-level-console=warn
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
protocol-timeout=60
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[global:backup]
archive-copy=y
start-fast=y
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
+ supplemental file: /backup/db/[BACKUP-DIFF-5]/backup.manifest
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
---------------------------------------------------------------
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
[backup]
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
backup-label="[BACKUP-DIFF-5]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
backup-prior="[BACKUP-FULL-3]"
backup-timestamp-copy-start=[TIMESTAMP]
backup-timestamp-start=[TIMESTAMP]
backup-timestamp-stop=[TIMESTAMP]
backup-type="diff"
[backup:db]
db-catalog-version=201409291
db-control-version=942
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-version="9.4"
[backup:option]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
option-archive-check=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-archive-copy=true
option-backup-standby=false
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
option-buffer-size=[BUFFER-SIZE]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-checksum-page=false
option-compress=true
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-compress-level=3
option-compress-level-network=1
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
option-compress-type="lz4"
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
option-delta=false
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
option-hardlink=false
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
option-online=false
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
option-process-max=2
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[backup:target]
pg_data={"path":"[TEST_PATH]/db-master/db/base-2/base","type":"path"}
pg_tblspc/2={"path":"../../tablespace/ts2","tablespace-id":"2","tablespace-name":"ts2","type":"link"}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-4]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[target:file]
pg_data/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/badchecksum.txt={"checksum":"f927212cd08d11a42a666b2f04235398e9ceeb51","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/12000={"checksum":"22c98d248ff548311eda88559e4a8405ed77c003","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/1/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","mode":"0660","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/17000={"checksum":"7579ada0808d7f98087a0a586d0df9de009cdc33","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/16384/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Fix incorrect offline upper bound for ignoring page checksum errors. For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX. This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode. Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 15:13:56 +02:00
pg_data/base/32768/33000={"checksum":"7a16d165e4775f7c92e8cdf60c0af57313f0bf90","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/33000.32767={"checksum":"6e99b589e550e68e934fd235ccba59fe5b592a9e","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/base/32768/33001={"checksum":"6bf316f11d28c28914ea9be92c00de9bea6d9a6b","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/32768/PG_VERSION={"checksum":"184473f470864e067ee3a22e64b47b0a1c356f29","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/base/base2.txt={"checksum":"cafac3c59553f2cfde41ce2e62e7662295f108c0","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
pg_data/changecontent.txt={"checksum":"a094d94583e209556d03c3c5da33131a065f1689","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
pg_data/changesize.txt={"checksum":"3905d5be2ec8d67f41435dab5e0dcda3ae47455d","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_data/changetime.txt={"checksum":"88087292ed82e26f3eb824d0bffc05ccf7a30f8d","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
pg_data/global/pg_control={"checksum":"4c77c900f7af0d9ab13fa9982051a42e0b637f6c","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/pg_stat/global.stat={"checksum":"e350d5ce0153f3e22d5db21cf2a4eff00f3ee877","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
pg_data/postgresql.conf={"checksum":"6721d92c9fcdf4248acff1f9a1377127d9064807","master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
pg_data/special-!_.*'()&!@;:+,?={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. Contributed by Cynthia Shang.
2018-08-24 22:50:33 +02:00
pg_data/zero_from_start={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
Add zero-length file to mock/all test.
2018-07-05 21:40:50 +02:00
pg_data/zerosize.txt={"master":true,"reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-2]}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2.txt={"checksum":"dc7f76e43c46101b47acc55ae4d593a9e6983578","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
pg_tblspc/2/[TS_PATH-1]/32768/tablespace2c.txt={"checksum":"dfcb8679956b734706cf87259d50c88f83e80e66","reference":"[BACKUP-FULL-3]","repo-size":[SIZE],"size":[SIZE],"timestamp":[TIMESTAMP-1]}
[target:file:default]
group="[GROUP-1]"
master=false
mode="0600"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[target:link]
pg_data/pg_tblspc/2={"destination":"../../tablespace/ts2"}
[target:link:default]
group="[GROUP-1]"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[target:path]
pg_data={}
pg_data/base={}
pg_data/base/1={}
pg_data/base/16384={}
pg_data/base/32768={}
pg_data/global={}
pg_data/pg_clog={}
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
pg_data/pg_log={}
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
pg_data/pg_stat={}
pg_data/pg_tblspc={}
pg_tblspc={}
pg_tblspc/2={}
pg_tblspc/2/[TS_PATH-1]={}
pg_tblspc/2/[TS_PATH-1]/32768={}
[target:path:default]
group="[GROUP-1]"
mode="0700"
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
user="[USER-1]"
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
Move info file checksum to the end of the file. Putting the checksum at the beginning of the file made it impossible to stream the file out when saving. The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning. Instead place the checksum at the end. This does not break the existing Perl or C code since the read is not order dependent. There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient. Reviewed by Cynthia Shang.
2019-08-22 01:45:48 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-3]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
[BACKUP-DIFF-4]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-3]","backup-reference":["[BACKUP-FULL-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-DIFF-5]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-prior":"[BACKUP-FULL-3]","backup-reference":["[BACKUP-FULL-3]"],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"diff","db-id":1,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
[db]
db-catalog-version=201409291
db-control-version=942
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
db-version="9.4"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Reference in New Issue Copy Permalink