db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Code Issues Releases Activity
cc9d7315db
pgbackrest/test/expect/mock-stanza-002.log

635 lines
39 KiB
Plaintext
Raw Normal View History

Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
run 002 - remote 1, s3 1, enc 1, cmp gz
=======================================
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
stanza-create db - fail on missing control file (backup host)
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online --log-level-file=[LOG-LEVEL-FILE] stanza-create
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-create command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 ERROR: [055]: raised from remote-0 protocol on 'db-master': unable to open missing file '[TEST_PATH]/db-master/db/base/global/pg_control' for read
P00 INFO: stanza-create command end: aborted with exception [055]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
stanza-upgrade db - fail on stanza not initialized since archive.info is missing (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-upgrade
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-upgrade command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 ERROR: [055]: unable to load info file '/archive/db/archive.info' or '/archive/db/archive.info.copy':
FileMissingError: unable to open '/archive/db/archive.info': No such file or directory
FileMissingError: unable to open '/archive/db/archive.info.copy': No such file or directory
HINT: archive.info cannot be opened but is required to push/get WAL segments.
HINT: is archive_command configured correctly in postgresql.conf?
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
HINT: has a stanza-create been performed?
HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: stanza-upgrade command end: aborted with exception [055]
stanza-create db - successfully create the stanza (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-create
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-create command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: stanza-create command end: completed successfully
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201306121
db-control-version=937
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Improve stanza-create command so that it does not error when the stanza already exists. Contributed by Cynthia Shang.
2018-02-01 21:12:03 +02:00
stanza-create db - do not fail on rerun of stanza-create - info files exist and DB section ok (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-create
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-create command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 INFO: stanza 'db' already exists and is valid
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Improve stanza-create command so that it does not error when the stanza already exists. Contributed by Cynthia Shang.
2018-02-01 21:12:03 +02:00
P00 INFO: stanza-create command end: completed successfully
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201306121
db-control-version=937
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
stanza-create db - fail on database mismatch and warn force option deprecated (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online --force stanza-create
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-create command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --force --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 WARN: option --force is no longer supported
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
P00 ERROR: [028]: backup and archive info files exist but do not match the database
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
HINT: is this the correct stanza?
HINT: did an error occur during stanza-upgrade?
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: stanza-create command end: aborted with exception [028]
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201306121
db-control-version=937
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
stanza-upgrade db - already up to date (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-upgrade
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-upgrade command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 INFO: stanza 'db' is already up to date
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
P00 INFO: stanza-upgrade command end: completed successfully
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201306121
db-control-version=937
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-id=1
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.3"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-push [TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001
------------------------------------------------------------------------------------------------------------------------------------
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: archive-push command begin [BACKREST-VERSION]: [[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001] --buffer-size=[BUFFER-SIZE] --compress-level=3 --compress-level-network=1 --config=[TEST_PATH]/db-master/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-host=backup --repo1-host-cmd=[BACKREST-BIN] --repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf --repo1-host-user=[USER-1] --stanza=db
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 15:26:33 +02:00
P00 INFO: pushed WAL file '000000010000000100000001' to the archive
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
P00 INFO: archive-push command end: completed successfully
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-push [TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000002
------------------------------------------------------------------------------------------------------------------------------------
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: archive-push command begin [BACKREST-VERSION]: [[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000002] --buffer-size=[BUFFER-SIZE] --compress-level=3 --compress-level-network=1 --config=[TEST_PATH]/db-master/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-host=backup --repo1-host-cmd=[BACKREST-BIN] --repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf --repo1-host-user=[USER-1] --stanza=db
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 15:26:33 +02:00
P00 INFO: pushed WAL file '000000010000000100000002' to the archive
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
P00 INFO: archive-push command end: completed successfully
Rename archive-queue-max option to archive-push-queue-max. This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-19 01:37:35 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --log-level-console=warn --archive-push-queue-max=33554432 --stanza=db archive-push [TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 15:26:33 +02:00
P00 ERROR: [044]: PostgreSQL version 9.4, system-id 1000000000000000094 do not match stanza version 9.3, system-id 1000000000000000093
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
HINT: are you archiving to the correct stanza?
stanza-upgrade db - successful upgrade creates additional history (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-upgrade
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-upgrade command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: stanza-upgrade command end: completed successfully
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201409291
db-control-version=942
db-id=2
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.4"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
2={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-id=2
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000094
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.4"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
2={"db-id":1000000000000000094,"db-version":"9.4"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --stanza=db archive-get 000000010000000100000002 [TEST_PATH]/db-master/db/base/pg_xlog/RECOVERYXLOG
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: archive-get command begin [BACKREST-VERSION]: [000000010000000100000002, [TEST_PATH]/db-master/db/base/pg_xlog/RECOVERYXLOG] --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/db-master/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/db-master/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/db-master/log[] --no-log-timestamp --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-host=backup --repo1-host-cmd=[BACKREST-BIN] --repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf --repo1-host-user=[USER-1] --stanza=db
Make archive-get info messages consistent between C and Perl implementations. The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all. Temporarily track the async server status with a flag so that info messages are not output in the async process. The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 18:30:48 +02:00
P00 INFO: found 000000010000000100000002 in the archive
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
P00 INFO: archive-get command end: completed successfully
Rename archive-queue-max option to archive-push-queue-max. This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-19 01:37:35 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --log-level-console=warn --archive-push-queue-max=33554432 --stanza=db archive-push [TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
full backup - create first full backup (backup host)
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=2 --no-online --type=full --stanza=db backup
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: backup command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level=3 --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-retention-full=2 --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db --start-fast --type=full
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/RECOVERYXLOG (16MB, 33%) checksum 488ba4b8b98acc510bce86b8f16e3c1ed9886a29
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000002 (16MB, 66%) checksum 488ba4b8b98acc510bce86b8f16e3c1ed9886a29
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001 (16MB, 99%) checksum e40de8cea99dd469c3efb47f5f33a73c7390fb9c
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/global/pg_control (8KB, 100%) checksum 4c77c900f7af0d9ab13fa9982051a42e0b637f6c
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/archive_status/000000010000000100000002.ready (0B, 100%)
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/archive_status/000000010000000100000001.ready (0B, 100%)
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
P00 INFO: full backup size = 48MB
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: new backup label = [BACKUP-FULL-1]
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: backup command end: completed successfully
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/backup/pgbackrest.conf --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-retention-full=2 --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
P00 INFO: remove archive path: /archive/db/9.3-1
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: full backup total < 2 - using oldest full backup for 9.4-2 archive retention
The expire command is implemented entirely in C. This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests. Along the way at least one bug was fixed, see issue #748. Contributed by Cynthia Shang.
2019-06-18 21:19:20 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: expire command end: completed successfully
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
repo1-host-user=[USER-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
spool-path=[TEST_PATH]/db-master/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-master/db/base
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global:backup]
archive-copy=y
start-fast=y
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
stanza-upgrade db - successfully upgrade (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-upgrade
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-upgrade command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 INFO: stanza-upgrade command end: completed successfully
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":2,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
cipher-pass=[CIPHER-PASS-1]
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
db-catalog-version=201510051
db-control-version=942
db-id=3
db-system-id=1000000000000000095
db-version="9.5"
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db:history]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
2={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
3={"db-catalog-version":201510051,"db-control-version":942,"db-system-id":1000000000000000095,"db-version":"9.5"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
db-id=3
db-system-id=1000000000000000095
db-version="9.5"
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
2={"db-id":1000000000000000094,"db-version":"9.4"}
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
3={"db-id":1000000000000000095,"db-version":"9.5"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
stanza-upgrade db - upgrade fails with mismatched db-ids (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db --no-online stanza-upgrade
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-upgrade command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 ERROR: [028]: backup info file and archive info file do not match
archive: id = 2, version = 9.5, system-id = 1000000000000000095
backup : id = 3, version = 9.5, system-id = 1000000000000000095
HINT: this may be a symptom of repository corruption!
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
P00 INFO: stanza-upgrade command end: aborted with exception [028]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /backup/db/backup.info
-------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
[backup:current]
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
[BACKUP-FULL-1]={"backrest-format":5,"backrest-version":"[VERSION-1]","backup-archive-start":null,"backup-archive-stop":null,"backup-info-repo-size":[SIZE],"backup-info-repo-size-delta":[DELTA],"backup-info-size":[SIZE],"backup-info-size-delta":[DELTA],"backup-timestamp-start":[TIMESTAMP],"backup-timestamp-stop":[TIMESTAMP],"backup-type":"full","db-id":2,"option-archive-check":false,"option-archive-copy":true,"option-backup-standby":false,"option-checksum-page":false,"option-compress":true,"option-hardlink":false,"option-online":false}
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
db-catalog-version=201510051
db-control-version=942
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
db-id=3
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
db-system-id=1000000000000000095
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-version="9.5"
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-catalog-version":201306121,"db-control-version":937,"db-system-id":1000000000000000093,"db-version":"9.3"}
2={"db-catalog-version":201409291,"db-control-version":942,"db-system-id":1000000000000000094,"db-version":"9.4"}
3={"db-catalog-version":201510051,"db-control-version":942,"db-system-id":1000000000000000095,"db-version":"9.5"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
+ supplemental file: /archive/db/archive.info
---------------------------------------------
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[backrest]
backrest-format=5
backrest-version="[VERSION-1]"
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
[cipher]
cipher-pass=[CIPHER-PASS-2]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db]
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
db-id=1
db-system-id=1000000000000000093
db-version="9.3"
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[db:history]
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
1={"db-id":1000000000000000093,"db-version":"9.3"}
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 19:48:28 +02:00
[backrest]
backrest-checksum="[CHECKSUM]"
Rename archive-queue-max option to archive-push-queue-max. This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-19 01:37:35 +02:00
> [CONTAINER-EXEC] db-master [BACKREST-BIN] --config=[TEST_PATH]/db-master/pgbackrest.conf --log-level-console=warn --archive-push-queue-max=33554432 --stanza=db archive-push [TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
diff backup - diff changed to full backup (backup host)
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --repo1-retention-full=2 --no-online --type=diff --stanza=db backup
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: backup command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level=3 --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --no-online --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-retention-full=2 --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db --start-fast --type=diff
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 WARN: no prior backup exists, diff backup has been changed to full
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/RECOVERYXLOG (16MB, 33%) checksum 488ba4b8b98acc510bce86b8f16e3c1ed9886a29
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000002 (16MB, 66%) checksum 488ba4b8b98acc510bce86b8f16e3c1ed9886a29
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/000000010000000100000001 (16MB, 99%) checksum 65b64687e7bfd1a918c6144f84a133f03e50ac57
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/global/pg_control (8KB, 100%) checksum 4969435f3b36bfaa0f5a486bef97f1988a135520
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/archive_status/000000010000000100000002.ready (0B, 100%)
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P01 INFO: backup file db-master:[TEST_PATH]/db-master/db/base/pg_xlog/archive_status/000000010000000100000001.ready (0B, 100%)
Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. Fixed by Cynthia Shang. Reported by Clinton Adams.
2017-11-17 00:18:51 +02:00
P00 INFO: full backup size = 48MB
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: new backup label = [BACKUP-FULL-2]
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: backup command end: completed successfully
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: expire command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --config=[TEST_PATH]/backup/pgbackrest.conf --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-retention-full=2 --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
The expire command is implemented entirely in C. This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests. Along the way at least one bug was fixed, see issue #748. Contributed by Cynthia Shang.
2019-06-18 21:19:20 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
P00 INFO: expire command end: completed successfully
+ supplemental file: [TEST_PATH]/db-master/pgbackrest.conf
----------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-path=[TEST_PATH]/db-master/db/base
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-timeout=45
lock-path=[TEST_PATH]/db-master/lock
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/db-master/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-host=backup
repo1-host-cmd=[BACKREST-BIN]
repo1-host-config=[TEST_PATH]/backup/pgbackrest.conf
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
repo1-host-user=[USER-1]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
spool-path=[TEST_PATH]/db-master/spool
+ supplemental file: [TEST_PATH]/backup/pgbackrest.conf
-------------------------------------------------------
[db]
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
pg1-host=db-master
pg1-host-cmd=[BACKREST-BIN]
pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf
pg1-host-user=[USER-1]
pg1-path=[TEST_PATH]/db-master/db/base
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global]
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
buffer-size=[BUFFER-SIZE]
Fix custom settings for compress-level option being ignored. Reported by Jens Wilke.
2017-10-23 00:05:46 +02:00
compress-level=3
compress-level-network=1
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
db-timeout=45
lock-path=[TEST_PATH]/backup/lock
Reduce expect log level in mock/stanza tests. The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior. Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit. Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
log-level-console=detail
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-level-file=[LOG-LEVEL-FILE]
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
log-level-stderr=off
log-path=[TEST_PATH]/backup/log
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
log-subprocess=[LOG-SUBPROCESS]
Set log-timestamp=n for integration tests. This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 17:24:38 +02:00
log-timestamp=n
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
protocol-timeout=60
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
repo1-cipher-pass=x
repo1-cipher-type=aes-256-cbc
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
repo1-path=/
repo1-s3-bucket=pgbackrest-dev
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-key=accessKey1
repo1-s3-key-secret=verySecretKey1
repo1-s3-region=us-east-1
repo1-s3-verify-ssl=n
repo1-type=s3
Consolidate stanza-create and stanza-upgrade tests into new stanza test. Contributed by Cynthia Shang.
2017-08-04 22:10:51 +02:00
[global:backup]
archive-copy=y
start-fast=y
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
stanza-delete db - fail on missing stop file (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db stanza-delete
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-delete command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
P00 ERROR: [055]: stop file does not exist for stanza 'db'
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
HINT: has the pgbackrest stop command been run on this server for this stanza?
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
P00 INFO: stanza-delete command end: aborted with exception [055]
db must not exist for successful delete
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
> list backup
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
db
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
db must not exist for successful delete
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
> list archive
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
db
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
stop db stanza (backup host)
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db stop
------------------------------------------------------------------------------------------------------------------------------------
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
P00 INFO: stop command begin [BACKREST-VERSION]: --config=[TEST_PATH]/backup/pgbackrest.conf --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
P00 INFO: stop command end: completed successfully
stanza-delete db - successfully delete the stanza (backup host)
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
> [CONTAINER-EXEC] backup [BACKREST-BIN] --config=[TEST_PATH]/backup/pgbackrest.conf --stanza=db stanza-delete
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.
2020-02-27 19:25:32 +02:00
P00 INFO: stanza-delete command begin [BACKREST-VERSION]: --buffer-size=[BUFFER-SIZE] --compress-level-network=1 --config=[TEST_PATH]/backup/pgbackrest.conf --db-timeout=45 --lock-path=[TEST_PATH]/backup/lock --log-level-console=detail --log-level-file=[LOG-LEVEL-FILE] --log-level-stderr=off --log-path=[TEST_PATH]/backup/log[] --no-log-timestamp --pg1-host=db-master --pg1-host-cmd=[BACKREST-BIN] --pg1-host-config=[TEST_PATH]/db-master/pgbackrest.conf --pg1-host-user=[USER-1] --pg1-path=[TEST_PATH]/db-master/db/base --protocol-timeout=60 --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=pgbackrest-dev --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-1 --no-repo1-s3-verify-tls --repo1-type=s3 --stanza=db
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
P00 DETAIL: tls statistics:[TLS-STATISTICS]
P00 INFO: http statistics:[HTTP-STATISTICS]
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
P00 INFO: stanza-delete command end: completed successfully
db must not exist for successful delete
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
> list backup
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
db must not exist for successful delete
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
> list archive
Add stanza-delete command to cleanup unused stanzas. Contributed by Cynthia Shang.
2018-01-03 19:23:33 +02:00
------------------------------------------------------------------------------------------------------------------------------------
Reference in New Issue Copy Permalink