db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Code Issues Releases Activity
b64755d635
pgbackrest/test/define.yaml

736 lines
25 KiB
YAML
Raw Normal View History

Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# **********************************************************************************************************************************
# Test Definition
#
# Contains definitions for all unit and integration tests.
#
# The definitions consist of modules and tests. Modules group together tests that are related and allow common settings for all the
# tests. Every module and test must have a name. Tests must also have a total. If this total does not match the actual number of
# runs in a test then an error will be thrown.
#
# Most options can be set for modules and tests (test option will override module option if both are set):
# * db - determines if the test will be run against multiple db versions
# * coverage - determines coverage for the test
# full - the module/test provides full coverage for the code module
# partial - the module/test provides partial coverage for the code module
# noCode - the code module should not contain any coverable code. If it does an error will be thrown.
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
# * define - defines for C code (will also be applied to the test harness)
# * define-test - defines for the test harness
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# * debugUnitSuppress - don't define DEBUG_UNIT for unit tests -- this is used to test unit test debugging macros
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
# * binReq - is the pgbackrest binary required for this test?
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
# * containerReq - is this test required to run in a container?
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
#
# Some options are unique to tests:
# * total - total runs in the test
# * vm - VMs that the test will be run on
Allow modules to be included for testing without requiring coverage. Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested. The include directive allows this. Update modules that had previously been added to coverage that only need to be included.
2019-07-26 02:15:06 +02:00
# * include - modules to include directly into test.c (all files in coverage are automatically included)
# This is useful when a module's internal data needs to be manipulated for testing but no coverage is added by the test.
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# **********************************************************************************************************************************
Divide tests into three types (unit, integration, performance). Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual. Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 15:12:25 +02:00
# **********************************************************************************************************************************
# Unit tests
# **********************************************************************************************************************************
unit:
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: common
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: error
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 15:38:09 +02:00
total: 8
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_ERROR -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/error: full
common/error.auto: noCode
# ----------------------------------------------------------------------------------------------------------------------------
- name: assert-on
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
total: 1
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/assert: noCode
# ----------------------------------------------------------------------------------------------------------------------------
- name: assert-off
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
total: 1
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define: -DNDEBUG
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
debugUnitSuppress: true
coverage:
common/assert: noCode
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-13 02:42:26 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
- name: stack-trace
total: 4
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-13 02:42:26 +02:00
coverage:
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
common/stackTrace: full
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-13 02:42:26 +02:00
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: mem-context
total: 7
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_MEM_CONTEXT -DNO_LOG
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/memContext: full
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: time
Add basic time management functions. These are similar to what mktime() and strptime() do but they ignore the local system timezone which saves having to munge the TZ env variable to do time conversions.
2020-01-07 00:18:52 +02:00
total: 3
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_ERROR -DNO_LOG
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
coverage:
common/time: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: fork
total: 1
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_LOG
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
coverage:
common/fork: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: log
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 15:38:09 +02:00
total: 5
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define-test: -DNO_LOG
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/log: full
# ----------------------------------------------------------------------------------------------------------------------------
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-21 00:51:42 +02:00
- name: debug-off
total: 2
Improve single test run performance. Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes). This simplifies some of the rsync logic and allows make to do what it does best. Also split build flag files into test, harness, and build to reduce rebuilds. Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.
2018-11-03 22:34:04 +02:00
define: -DNDEBUG
define-test: -DNO_LOG
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-21 00:51:42 +02:00
debugUnitSuppress: true
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
common/debug: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-21 00:51:42 +02:00
- name: debug-on
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
common/debug: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
Add macros for object free functions. Most of the *Free() functions are pretty generic so add macros to make creating them as easy as possible. Create a distinction between *Free() functions that the caller uses to free memory and callbacks that free third-party resources. There are a number of cases where a driver needs to free resources but does not need a normal *Free() because it is handled by the interface. Add common/object.h for macros that make object maintenance easier. This pattern can also be used for many more object functions.
2019-05-04 00:52:54 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: object
total: 1
coverage:
common/object: noCode
Improved lock implementation written in C. Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 15:36:12 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: lock
total: 2
coverage:
common/lock: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: exit
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-13 02:42:26 +02:00
total: 3
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/exit: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: wait
total: 1
coverage:
common/wait: full
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-convert
Add TIME parameter debug type. Previously we were using int64_t to debug time_t but this may not be right depending on how the compiler represents time_t, e.g. it could be a float. Since a mismatch would have caused a compiler error we are not worried that this has actually happened, and anyway the worst case is that the debug log would be wonky. The primary benefit, aside from correctness, is that it makes choosing a parameter debug type for time_t obvious.
2019-11-08 16:46:00 +02:00
total: 10
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 17:57:32 +02:00
coverage:
common/type/convert: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-list
Use bsearch() on sorted lists rather than an iterative method. bsearch() is far more efficient than an iterative approach except in the most trivial cases. For now insert will reset the sort order to none and the list will need to be resorted before bsearch() can be used. This is necessary because item pointers are not stable after a sort, i.e. they can move around. Until lists are stable it's not a good idea to surprise the caller by mixing up their pointers on insert.
2019-09-28 16:08:20 +02:00
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/type/list: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-string
Add strLstMergeAnti() for merge anti-joins. We deal with some pretty big lists in archive-push so a nested-loop anti-join looked like it would not be efficient enough. This merge anti-join should do the trick even though both lists must be sorted first.
2019-03-25 18:35:20 +02:00
total: 26
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/type/string: full
common/type/stringList: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-buffer
Add bufHex()to Buffer object. A general-purpose function for converting buffers to hex strings.
2018-09-26 23:33:48 +02:00
total: 6
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/type/buffer: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-variant
Add unsigned int Variant type. This is better than using (unsigned int)varUInt64() because bounds checking is performed.
2019-04-19 17:22:43 +02:00
total: 13
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/type/variant: full
common/type/variantList: full
Add MostCommonValue object. Calculate the most common value in a list of variants. If there is a tie then the first value passed to mcvUpdate() wins. mcvResult() can be called multiple times because it does not end processing, but there is a cost to calculating the result each time since it is not stored.
2019-08-19 02:46:34 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-mcv
total: 1
coverage:
common/type/mcv: full
Add basic C JSON parser.
2018-08-09 14:06:23 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-json
Add separate functions to encode/decode each JSON type. In most cases the JSON type is known so this is more efficient than converting to Variant first, both in terms of memory and time. Also rename some of the existing functions for consistency.
2019-04-23 00:41:01 +02:00
total: 11
Add basic C JSON parser.
2018-08-09 14:06:23 +02:00
coverage:
common/type/json: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-key-value
total: 2
coverage:
common/type/keyValue: full
Implement filters that do not modify the buffer. Update cryptoHash to use the new interface.
2018-07-25 03:08:27 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Add interface objects for libxml2. Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2. This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-21 03:40:11 +02:00
- name: type-xml
total: 1
coverage:
common/type/xml: full
Add user module for managing system users/groups. Centralize the management of users and groups. Also update Posix storage driver where users/groups were already in use.
2019-09-09 02:11:51 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: user
total: 1
coverage:
common/user: full
Add interface objects for libxml2. Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2. This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.
2018-11-21 03:40:11 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Implement filters that do not modify the buffer. Update cryptoHash to use the new interface.
2018-07-25 03:08:27 +02:00
- name: io
total: 4
coverage:
common/io/bufferRead: full
common/io/bufferWrite: full
Filters can now produce output that differs from input. This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 20:21:53 +02:00
common/io/filter/buffer: full
Implement filters that do not modify the buffer. Update cryptoHash to use the new interface.
2018-07-25 03:08:27 +02:00
common/io/filter/filter: full
common/io/filter/group: full
Add IoSink filter. Discard all data passed to the filter. Useful for calculating size/checksum on a remote system when no data needs to be returned. Update ioReadDrain() to automatically use the IoSink filter.
2019-07-18 14:42:42 +02:00
common/io/filter/sink: full
Filters can now produce output that differs from input. This allows filters such as compression, encryption, etc. to be implemented.
2018-08-14 20:21:53 +02:00
common/io/filter/size: full
Add IoHandleRead and IoHandleWrite objects. General i/o objects for reading and writing file descriptors, in particular those that can block. In other words, these are not generally to be used with file descriptors for actual files, but rather pipes, sockets, etc.
2019-01-17 22:08:31 +02:00
common/io/handleRead: full
Rename common/io/handle module to common/io/handleWrite. ioHandleWriteOneStr() will become a helper function for the IoHandleWrite object.
2019-01-06 14:37:39 +02:00
common/io/handleWrite: full
Implement filters that do not modify the buffer. Update cryptoHash to use the new interface.
2018-07-25 03:08:27 +02:00
common/io/io: full
common/io/read: full
common/io/write: full
Add TlsClient object. A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed. This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session. Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-22 01:43:25 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: io-tls
Updates for openssl 1.1.1. Some HTTP error tests were failing after the upgrade to openssl 1.1.1, though the rest of the unit and integration tests worked fine. This seemed to be related to the very small messages used in the error testing, but it pointed to an issue with the code not being fully compliant, made worse by auto-retry being enabled by default. Disable auto-retry and implement better error handling to bring the code in line with openssl recommendations. There's no evidence this is a problem in the field, but having all the tests pass seems like a good idea and the new code is certainly more robust. Coverage will be complete in the next commit when openssl 1.1.1 is introduced.
2019-07-03 04:09:12 +02:00
total: 4
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
Add TlsClient object. A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed. This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session. Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().
2018-11-22 01:43:25 +02:00
coverage:
common/io/tls/client: full
Add HttpClient object. A robust HTTP client with pipelining support and automatic retries. Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried. Only the HTTPS protocol is currently supported.
2018-11-22 02:11:45 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: io-http
Add httpLastModifiedToTime() to parse HTTP last-modified header.
2020-01-07 00:24:49 +02:00
total: 6
Add HttpClient object. A robust HTTP client with pipelining support and automatic retries. Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried. Only the HTTPS protocol is currently supported.
2018-11-22 02:11:45 +02:00
coverage:
Add HTTP client cache. This cache manages multiple http clients and returns one to the caller that is not busy. It is the responsibility of the caller to indicate when they are done with a client. If returnContent is set then the client will automatically be marked done. Also add special handing for HEAD requests to recognize that content-length is informational only and no content is expected.
2019-06-11 16:48:22 +02:00
common/io/http/cache: full
Add HttpClient object. A robust HTTP client with pipelining support and automatic retries. Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried. Only the HTTPS protocol is currently supported.
2018-11-22 02:11:45 +02:00
common/io/http/client: full
common/io/http/common: full
common/io/http/header: full
common/io/http/query: full
Move compress module to common/compress. It makes sense for the compression code to be in common since it is not pgBackRest-specific.
2019-03-10 13:11:20 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Rename "gzip" to "gz". "gz" was used as the extension but "gzip" was generally used for function and type naming. With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code. Since the extension is standard and we must use it, also use the extension for all naming.
2020-02-27 19:09:05 +02:00
- name: compress
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
total: 3
Move compress module to common/compress. It makes sense for the compression code to be in common since it is not pgBackRest-specific.
2019-03-10 13:11:20 +02:00
coverage:
Rename "gzip" to "gz". "gz" was used as the extension but "gzip" was generally used for function and type naming. With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code. Since the extension is standard and we must use it, also use the extension for all naming.
2020-02-27 19:09:05 +02:00
common/compress/gz/common: full
common/compress/gz/compress: full
common/compress/gz/decompress: full
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
common/compress/lz4/common: full
common/compress/lz4/compress: full
common/compress/lz4/decompress: full
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
common/compress/helper: full
Move compress module to common/compress. It makes sense for the compression code to be in common since it is not pgBackRest-specific.
2019-03-10 13:11:20 +02:00
Move crypto module to common/crypto. It makes sense for the crypto code to be in common since it is not pgBackRest-specific. Also combine the crypto tests into a single module.
2019-03-10 13:27:30 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: crypto
total: 3
coverage:
common/crypto/cipherBlock: full
common/crypto/common: full
common/crypto/hash: full
Add Exec object. Executes a child process and allows the calling process to communicate with it using read/write io. This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose execution.
2019-01-18 11:45:40 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: exec
total: 1
coverage:
common/exec: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: encode
total: 1
coverage:
common/encode: full
common/encode/base64: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: reg-exp
Add regExpPrefix() to aid in static prefix searches. The storageList() command accepts a regular expression as a filter. This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code. However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally. S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible. Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever. It requires a ^ anchor and stops scanning when the first special character is found.
2018-11-09 23:50:22 +02:00
total: 3
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/regExp: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: ini
Move C performance tests out of unit tests. Performance tests were being done in unit tests until there was a better place to put them. Now there is, so move them there.
2019-09-28 20:24:27 +02:00
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
common/ini: full
# ********************************************************************************************************************************
- name: postgres
test:
Add PostgreSQL query client. This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts. Testing is performed using a shim that can use scripted responses to test all aspects of the client code. The shim will be very useful for testing backup scenarios on complex topologies. Reviewed by Cynthia Shang.
2019-07-25 20:50:02 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: client
total: 1
coverage:
postgres/client: full
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 17:20:51 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
- name: interface
Move logic from postgres/pageChecksum to command/backup/pageChecksum(). The postgres/pageChecksum module was designed as an interface to the C structs for the Perl code. The new C code can do this directly so no need for an interface. Move the remaining test for pgPageChecksum() into the postgres/interface test module.
2020-03-05 23:12:54 +02:00
total: 9
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 17:20:51 +02:00
coverage:
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
postgres/interface: full
Move logic from postgres/pageChecksum to command/backup/pageChecksum(). The postgres/pageChecksum module was designed as an interface to the C structs for the Perl code. The new C code can do this directly so no need for an interface. Move the remaining test for pgPageChecksum() into the postgres/interface test module.
2020-03-05 23:12:54 +02:00
postgres/interface/page: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: config
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: define
total: 2
coverage:
config/define: full
config/define.auto: noCode
# ----------------------------------------------------------------------------------------------------------------------------
- name: config
Fix C code to recognize host:port format like Perl does. This was not an intentional feature in Perl, but it works, so it makes sense to implement the same syntax in C. This is a break from other places where a -port option is explicitly supplied, so it may make sense to support both styles going forward. This commit does not address that, however. Reported by Kyle Nevins.
2019-04-10 23:48:34 +02:00
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
config/config: full
config/config.auto: noCode
# ----------------------------------------------------------------------------------------------------------------------------
- name: parse
Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. Contributed by Cynthia Shang.
2018-04-25 21:46:05 +02:00
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
config/parse: full
config/parse.auto: noCode
Reduce variables extern'd by the common/log module in debug builds. These days it is better to include the module in define.yaml when we need to poke at the internal implementation. This doesn't quite work for the log test harness, so for now some variables will need to remain extern'd in debug builds.
2020-03-17 00:16:27 +02:00
include:
- common/log
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: load
Fix error after log file open failure when processing should continue. The C code was warning on failure and continuing but the Perl logging code was never updated with the same feature. Rather than add the feature to Perl, just disable file logging if the log file cannot be opened. Log files are always opened by C first, so this will eliminate the error in Perl. Reported by vthriller.
2018-10-25 15:58:25 +02:00
total: 4
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
config/load: full
Reduce variables extern'd by the common/log module in debug builds. These days it is better to include the module in define.yaml when we need to poke at the internal implementation. This doesn't quite work for the log test harness, so for now some variables will need to remain extern'd in debug builds.
2020-03-17 00:16:27 +02:00
include:
- common/log
Add cfgExecParam() to generate parameters for executing commands. Parameters for the local/remote commands are based on parameters that are passed to the current command. Generate parameters for the new command based on the intersection of parameters between the current command and the command to be executed.
2019-01-17 22:29:19 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: exec
total: 1
coverage:
config/exec: full
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: protocol
total: 1
coverage:
config/protocol: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: storage
test:
# ----------------------------------------------------------------------------------------------------------------------------
Add CIFS storage driver. This driver borrows heavily from the Posix driver. At this point the only difference is that CIFS does not allow explicit directory fsyncs so they need to be suppressed. At some point the CIFS diver will also omit link support. With the addition of this driver repository storage is now writable.
2019-03-14 11:28:33 +02:00
- name: cifs
total: 1
coverage:
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/cifs/storage: full
storage/posix/storage: full
Add CIFS storage driver. This driver borrows heavily from the Posix driver. At this point the only difference is that CIFS does not allow explicit directory fsyncs so they need to be suppressed. At some point the CIFS diver will also omit link support. With the addition of this driver repository storage is now writable.
2019-03-14 11:28:33 +02:00
storage/helper: full
Allow modules to be included for testing without requiring coverage. Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested. The include directive allows this. Update modules that had previously been added to coverage that only need to be included.
2019-07-26 02:15:06 +02:00
include:
- storage/storage
Add CIFS storage driver. This driver borrows heavily from the Posix driver. At this point the only difference is that CIFS does not allow explicit directory fsyncs so they need to be suppressed. At some point the CIFS diver will also omit link support. With the addition of this driver repository storage is now writable.
2019-03-14 11:28:33 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Merge all posix storage tests into a single unit. As we add storage drivers it's important to keep the tests for each completely separate. Rather than have three tests for each driver, standardize on having a single test unit for each driver.
2018-09-17 17:45:41 +02:00
- name: posix
Make storage and logging dry-run aware. Enhance dry-run support added in 2fa69af8 by forbidding writes in the storage layer and adding prefixes to log messages. The former will protect against mistakes in dry-run implementations and the latter will make it clear when a command was executed in dry-run mode. Update expire unit tests with the new log prefix.
2020-03-16 23:24:21 +02:00
total: 21
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
coverage:
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/posix/read: full
storage/posix/storage: full
storage/posix/write: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
storage/helper: full
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/read: full
Merge all posix storage tests into a single unit. As we add storage drivers it's important to keep the tests for each completely separate. Rather than have three tests for each driver, standardize on having a single test unit for each driver.
2018-09-17 17:45:41 +02:00
storage/storage: full
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/write: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
Add remote storage objects. This is a partial implementation of remote storage with just enough functionality to get the info command working. The client is written in C but the server is still in Perl, which limits progress until a C server is written.
2019-01-18 22:04:37 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: remote
Add storageInfoList() support to remote storage driver.
2019-11-17 00:47:42 +02:00
total: 12
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
Add remote storage objects. This is a partial implementation of remote storage with just enough functionality to get the info command working. The client is written in C but the server is still in Perl, which limits progress until a C server is written.
2019-01-18 22:04:37 +02:00
coverage:
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/remote/read: full
storage/remote/protocol: full
storage/remote/storage: full
storage/remote/write: full
Add remote storage objects. This is a partial implementation of remote storage with just enough functionality to get the info command working. The client is written in C but the server is still in Perl, which limits progress until a C server is written.
2019-01-18 22:04:37 +02:00
storage/helper: full
storage/storage: full
Allow modules to be included for testing without requiring coverage. Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested. The include directive allows this. Update modules that had previously been added to coverage that only need to be included.
2019-07-26 02:15:06 +02:00
include:
- storage/read
- storage/write
Add S3 storage driver. Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-22 02:32:49 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: s3
total: 3
coverage:
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 21:46:15 +02:00
storage/s3/read: full
storage/s3/storage: full
storage/s3/write: full
Add S3 storage driver. Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.
2018-11-22 02:32:49 +02:00
storage/helper: full
storage/storage: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: protocol
test:
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: protocol
Add protocolStorageType*() to manage protocol storage types. Abstract the string representation of storage types that are passed over the protocol layer.
2019-11-23 17:22:11 +02:00
total: 9
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
coverage:
protocol/client: full
Add ProtocolCommand object. This formalizes the creation of protocol commands, which was previously done by creating KeyValue objects manually.
2019-02-27 19:48:30 +02:00
protocol/command: full
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
protocol/helper: full
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
protocol/parallel: full
protocol/parallelJob: full
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
protocol/server: full
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: info
test:
Migrate minimum set of code for reading archive.info files from Perl to C. Contributed by Cynthia Shang.
2018-08-09 14:57:21 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: info
Allow Info* objects to be created from scratch in C. Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C. The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created. Also, add functions needed to initialize each Info* object to a valid state. Contributed by Cynthia Shang.
2019-08-21 21:12:00 +02:00
total: 3
Migrate minimum set of code for reading archive.info files from Perl to C. Contributed by Cynthia Shang.
2018-08-09 14:57:21 +02:00
coverage:
info/info: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: info-pg
Allow Info* objects to be created from scratch in C. Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C. The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created. Also, add functions needed to initialize each Info* object to a valid state. Contributed by Cynthia Shang.
2019-08-21 21:12:00 +02:00
total: 2
Migrate minimum set of code for reading archive.info files from Perl to C. Contributed by Cynthia Shang.
2018-08-09 14:57:21 +02:00
coverage:
info/infoPg: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: info-archive
Add infoArchiveIdHistoryMatch() to the InfoArchive object. Match a PostgreSQL system identifier and version to a pgBackRest archive id. Contributed by Cynthia Shang.
2018-12-11 01:45:57 +02:00
total: 2
Migrate minimum set of code for reading archive.info files from Perl to C. Contributed by Cynthia Shang.
2018-08-09 14:57:21 +02:00
coverage:
info/infoArchive: full
Migrate control functions to detect stop files to C from Perl. Basic functions to detect the presence of stanza or all stop files and error when they are present. The functionality to detect stop files without error was not migrated. This functionality is only used by stanza-delete and will be migrated with that command.
2018-09-07 17:03:05 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
- name: info-backup
Improve and centralize backup dependency calculation. Add functions to select a current backup by label and to retrieve a backup dependency list for any given backup. Update the expire code to utilize the new functions and to expire backup sets from newest dependency to oldest.
2020-03-26 20:05:40 +02:00
total: 3
Migrate control functions to detect stop files to C from Perl. Basic functions to detect the presence of stanza or all stop files and error when they are present. The functionality to detect stop files without error was not migrated. This functionality is only used by stanza-delete and will be migrated with that command.
2018-09-07 17:03:05 +02:00
coverage:
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
info/infoBackup: full
Migrate control functions to detect stop files to C from Perl. Basic functions to detect the presence of stanza or all stop files and error when they are present. The functionality to detect stop files without error was not migrated. This functionality is only used by stanza-delete and will be migrated with that command.
2018-09-07 17:03:05 +02:00
Migrate backup manifest load/save to C. The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes, timestamps, etc. A list of databases is also included for selective restore. The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that nothing is missing or corrupt. It is also useful for reporting, e.g. size of backup, backup time, etc. For now, migrate enough functionality to implement the restore command. Reviewed by Cynthia Shang.
2019-09-23 19:50:46 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: manifest
Add functions for building new manifests. New manifests are built before a backup is performed. Reviewed by Cynthia Shang.
2019-12-09 01:43:47 +02:00
total: 6
Migrate backup manifest load/save to C. The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes, timestamps, etc. A list of databases is also included for selective restore. The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that nothing is missing or corrupt. It is also useful for reporting, e.g. size of backup, backup time, etc. For now, migrate enough functionality to implement the restore command. Reviewed by Cynthia Shang.
2019-09-23 19:50:46 +02:00
coverage:
info/manifest: full
Add Db object to encapsulate PostgreSQL queries and commands. Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check. Add the dbGet() helper function to get Db objects easily. Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together. Reviewed by Cynthia Shang.
2019-08-01 21:38:27 +02:00
# ********************************************************************************************************************************
- name: db
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: db
Add backup functions to Db object. These functions implement the database backup functionality for all supported versions.
2019-12-08 01:44:06 +02:00
total: 3
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
Add Db object to encapsulate PostgreSQL queries and commands. Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check. Add the dbGet() helper function to get Db objects easily. Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together. Reviewed by Cynthia Shang.
2019-08-01 21:38:27 +02:00
coverage:
db/db: full
db/helper: full
db/protocol: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
- name: command
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
test:
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 16:16:59 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
- name: archive-common
Move WAL path prefix logic into walPath(). This logic is used by both archive-push and archive-get.
2019-03-16 14:14:10 +02:00
total: 8
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 16:16:59 +02:00
coverage:
command/archive/common: full
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
- name: archive-get
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
total: 5
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 23:27:39 +02:00
coverage:
Consolidate archive async exec code. Move duplicated code to the common module. This will reduce copy and paste between the get and push modules when changes are made.
2020-02-11 06:30:43 +02:00
command/archive/common: full
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 21:42:31 +02:00
command/archive/get/file: full
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 23:27:39 +02:00
command/archive/get/get: full
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
command/archive/get/protocol: full
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 18:54:43 +02:00
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: archive-push
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 15:26:33 +02:00
total: 4
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
coverage:
Consolidate archive async exec code. Move duplicated code to the common module. This will reduce copy and paste between the get and push modules when changes are made.
2020-02-11 06:30:43 +02:00
command/archive/common: full
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 15:26:33 +02:00
command/archive/push/file: full
command/archive/push/protocol: full
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
command/archive/push/push: full
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 14:29:25 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: backup-common
Add backup type conversion functions. Convert back and forth between the string and enum representations of backup types.
2019-08-19 02:09:44 +02:00
total: 3
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 14:29:25 +02:00
coverage:
command/backup/common: full
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 13:52:03 +02:00
command/backup/pageChecksum: full
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 14:29:25 +02:00
The local command for backup is implemented entirely in C. The local process is now entirely migrated to C. Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C. Contributed by David Steele, Cynthia Shang.
2019-07-25 20:34:16 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: backup
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
total: 10
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
The local command for backup is implemented entirely in C. The local process is now entirely migrated to C. Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C. Contributed by David Steele, Cynthia Shang.
2019-07-25 20:34:16 +02:00
coverage:
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
command/backup/backup: full
The local command for backup is implemented entirely in C. The local process is now entirely migrated to C. Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C. Contributed by David Steele, Cynthia Shang.
2019-07-25 20:34:16 +02:00
command/backup/file: full
command/backup/protocol: full
Allow modules to be included for testing without requiring coverage. Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested. The include directive allows this. Update modules that had previously been added to coverage that only need to be included.
2019-07-26 02:15:06 +02:00
include:
Added redacted manifest to testBackupValidate(). The manifest is excellent for validation but including the entire manifest is too noisy and some values are architecture/algorithm dependent. Output a redacted version that contains the most important information which can be improved on over time.
2020-03-18 16:10:10 +02:00
- info/info
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
- info/manifest
Allow modules to be included for testing without requiring coverage. Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested. The include directive allows this. Update modules that had previously been added to coverage that only need to be included.
2019-07-26 02:15:06 +02:00
- storage/storage
The local command for backup is implemented entirely in C. The local process is now entirely migrated to C. Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C. Contributed by David Steele, Cynthia Shang.
2019-07-25 20:34:16 +02:00
The check command is implemented partly in C. Implement switch WAL and archive check in C but leave the rest in Perl for now. The main idea was to have some real integration tests for the new database code so the rest of the migration can wait. Reviewed by Cynthia Shang.
2019-08-02 02:35:01 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: check
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
total: 4
containerReq: true
The check command is implemented partly in C. Implement switch WAL and archive check in C but leave the rest in Perl for now. The main idea was to have some real integration tests for the new database code so the rest of the migration can wait. Reviewed by Cynthia Shang.
2019-08-02 02:35:01 +02:00
coverage:
Add checkDbConfig() to compare pgBackRest/PostgreSQL configs. Checking the PostgreSQL-reported path and version against the pgBackRest configuration helps ensure that pgBackRest is operating against the correct cluster. In Perl this functionality was in the Db object, but check seems like a better place for it in C. Contributed by Cynthia Shang.
2019-08-21 21:41:52 +02:00
command/check/common: full
The check command is implemented partly in C. Implement switch WAL and archive check in C but leave the rest in Perl for now. The main idea was to have some real integration tests for the new database code so the rest of the migration can wait. Reviewed by Cynthia Shang.
2019-08-02 02:35:01 +02:00
command/check/check: full
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: command
total: 1
coverage:
command/command: full
# ----------------------------------------------------------------------------------------------------------------------------
- name: control
The start/stop commands are implemented entirely in C. The Perl versions remain because they are still being used by the Perl stanza commands. Once the stanza commands are migrated they can be removed. Contributed by Cynthia Shang.
2019-08-09 21:17:18 +02:00
total: 3
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
coverage:
Rename control/control module to control/common. This is more consistent with how other common modules are named. Contributed by Cynthia Shang.
2019-07-31 17:35:58 +02:00
command/control/common: full
The start/stop commands are implemented entirely in C. The Perl versions remain because they are still being used by the Perl stanza commands. Once the stanza commands are migrated they can be removed. Contributed by Cynthia Shang.
2019-08-09 21:17:18 +02:00
command/control/start: full
command/control/stop: full
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
The expire command is implemented entirely in C. This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests. Along the way at least one bug was fixed, see issue #748. Contributed by Cynthia Shang.
2019-06-18 21:19:20 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: expire
Error on db history mismatch when expiring. Amend commit 434cd832 to error when the db history in archive.info and backup.info do not match. The Perl code would attempt to reconcile the history by matching on system id and version but we are not planning to migrate that code to C. It's possible that there are users with mismatches but if so they should have been getting errors from info for the last six months. It's easy enough to manually fix these files if there are any mismatches in the field. Contributed by Cynthia Shang.
2019-06-24 17:59:44 +02:00
total: 7
The expire command is implemented entirely in C. This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests. Along the way at least one bug was fixed, see issue #748. Contributed by Cynthia Shang.
2019-06-18 21:19:20 +02:00
coverage:
command/expire/expire: full
Refactor command/expire unit test module. Add titles and use a Buffer to store backup.info instead of a String.
2020-03-23 20:31:04 +02:00
include:
- info/infoBackup
Move help/help test module to command/help.
2019-03-10 11:55:01 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: help
total: 4
coverage:
command/help/help: full
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: info
total: 3
coverage:
command/info/info: full
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: local
total: 1
coverage:
command/local/local: full
Rename test modules for consistency. The conventions for command and info tests have shifted in the C modules, though not even all the C modules got the message.
2019-02-23 18:51:52 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: remote
total: 1
coverage:
command/remote/remote: full
The local command for restore is implemented entirely in C. This is just the part of restore run by the local helper processes, not the entire command. Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 23:07:37 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: restore
Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 23:38:05 +02:00
total: 12
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 18:06:30 +02:00
containerReq: true
pgBackRest is now pure C. Remove embedded Perl from the distributed binary. This includes code, configure, Makefile, and packages. The distributed binary is now pure C. Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory. Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule. Remove "c" option that allowed the remote to tell if it was being called from C or Perl. Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests. Update build and installation instructions in the user guide. Remove all Perl unit tests. Remove obsolete Perl code. In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally. Any remaining Perl code is required for testing, documentation, or code generation. Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test. This had been the actual meaning for quite some time but the key was never renamed.
2019-12-14 00:55:41 +02:00
binReq: true
The local command for restore is implemented entirely in C. This is just the part of restore run by the local helper processes, not the entire command. Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 23:07:37 +02:00
coverage:
command/restore/file: full
command/restore/protocol: full
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 13:52:02 +02:00
command/restore/restore: full
include:
- common/user
- info/infoBackup
- info/manifest
The local command for restore is implemented entirely in C. This is just the part of restore run by the local helper processes, not the entire command. Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.
2019-05-20 23:07:37 +02:00
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: stanza
total: 5
coverage:
command/stanza/common: full
command/stanza/create: full
command/stanza/upgrade: full
command/stanza/delete: full
Add ls command. Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging. This command is internal-only so the interface may change at any time without notice.
2019-05-28 16:03:48 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Rename ls command to repo-ls. This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands. Since the repo storage can be located lots of places having a common way to list it makes sense. Prefix with repo- to make the scope of this command clear. Update documentation to reflect this change.
2020-03-09 22:41:04 +02:00
- name: repo
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 23:15:03 +02:00
total: 4
Add ls command. Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging. This command is internal-only so the interface may change at any time without notice.
2019-05-28 16:03:48 +02:00
coverage:
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 23:15:03 +02:00
command/repo/get: full
Rename ls command to repo-ls. This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands. Since the repo storage can be located lots of places having a common way to list it makes sense. Prefix with repo- to make the scope of this command clear. Update documentation to reflect this change.
2020-03-09 22:41:04 +02:00
command/repo/ls: full
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 23:15:03 +02:00
command/repo/put: full
command/repo/rm: full
include:
# command/repo/create is currently for testing purposes only so coverage is not provided except in integration. In the
# future this will probably be rolled into a custom object store server implementation.
- command/repo/create
Add ls command. Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging. This command is internal-only so the interface may change at any time without notice.
2019-05-28 16:03:48 +02:00
Divide tests into three types (unit, integration, performance). Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual. Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 15:12:25 +02:00
# **********************************************************************************************************************************
# Integration tests
#
# Integration tests are not run in a container. They are expected to create their own containers since most integration runs will
# create more than one. For this reason each run is executed individually.
# **********************************************************************************************************************************
integration:
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: mock
test:
# ----------------------------------------------------------------------------------------------------------------------------
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
- name: stanza
Create test matrix for mock/stanza to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-24 07:42:41 +02:00
total: 2
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
- name: expire
Create test matrix for mock/expire to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests.
2019-03-01 19:04:26 +02:00
total: 2
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: archive
Create test matrix for mock/archive to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-02-23 15:59:39 +02:00
total: 2
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
- name: archive-stop
Create test matrix for mock/archive-stop to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-01 17:12:41 +02:00
total: 2
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
# ----------------------------------------------------------------------------------------------------------------------------
- name: all
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
total: 2
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: real
Move archive-stop and expire tests to the mock module. These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 18:57:02 +02:00
db: true
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: all
total: 6
Divide tests into three types (unit, integration, performance). Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual. Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 15:12:25 +02:00
# **********************************************************************************************************************************
# Performance tests
#
# Performance tests run in a single container but are more like integration tests than unit tests since they call the pgbackrest
# executable directly.
# **********************************************************************************************************************************
performance:
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
# ********************************************************************************************************************************
- name: performance
test:
# ----------------------------------------------------------------------------------------------------------------------------
Identify Perl performance test by appending -perl. This is intended to differentiate the upcoming C performance tests from the Perl performance tests that will eventually be migrated.
2019-09-28 19:17:21 +02:00
- name: archive-perl
Move test definitions to test/define.yaml. The location is better because it is no longer buried in the Perl test libs. Also, the data can be easily accessed from C.
2018-04-08 16:19:24 +02:00
total: 1
Move C performance tests out of unit tests. Performance tests were being done in unit tests until there was a better place to put them. Now there is, so move them there.
2019-09-28 20:24:27 +02:00
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: type
total: 3
Reference in New Issue Copy Permalink