db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Code Issues Releases Activity
cb891fa2d4
pgbackrest/CONTRIBUTING.md

719 lines
37 KiB
Markdown
Raw Normal View History

Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
# pgBackRest <br/> Contributing to pgBackRest
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
## Table of Contents
[Introduction](#introduction)
[Building a Development Environment](#building-a-development-environment)
[Coding](#coding)
[Testing](#testing)
[Submitting a Pull Request](#submitting-a-pull-request)
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
## Introduction
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
This documentation is intended to assist contributors to pgBackRest by outlining some basic steps and guidelines for contributing to the project.
Code fixes or new features can be submitted via pull requests. Ideas for new features and improvements to existing functionality or documentation can be [submitted as issues](https://github.com/pgbackrest/pgbackrest/issues). You may want to check the [Project Boards](https://github.com/pgbackrest/pgbackrest/projects) to see if your suggestion has already been submitted.
Bug reports should be [submitted as issues](https://github.com/pgbackrest/pgbackrest/issues). Please provide as much information as possible to aid in determining the cause of the problem.
You will always receive credit in the [release notes](http://www.pgbackrest.org/release.html) for your contributions.
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Coding standards are defined in [CODING.md](https://github.com/pgbackrest/pgbackrest/blob/main/CODING.md) and some important coding details and an example are provided in the [Coding](#coding) section below. At a minimum, unit tests must be written and run and the documentation generated before [submitting a Pull Request](#submitting-a-pull-request); see the [Testing](#testing) section below for details.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
## Building a Development Environment
Update test container OS versions. Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 23:29:20 +02:00
This example is based on Ubuntu 20.04, but it should work on many versions of Debian and Ubuntu.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
pgbackrest-dev => Install development tools
```
Remove custom lcov build from contributing documentation. Using the stock lcov no longer seems to be a problem, probably due to a better implementation in CoverageTest.pm.
2020-03-22 01:08:27 +02:00
sudo apt-get install rsync git devscripts build-essential valgrind lcov autoconf \
Add missing package to contributing documentation.
2020-03-21 15:51:12 +02:00
autoconf-archive libssl-dev zlib1g-dev libxml2-dev libpq-dev pkg-config \
v2.32: Repository Commands Bug Fixes: * Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.) Features: * Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.) * Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.) * Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.) Improvements: * Improve archive-get performance. (Reviewed by Cynthia Shang.)
2021-02-08 16:08:16 +02:00
libxml-checker-perl libyaml-perl libdbd-pg-perl liblz4-dev liblz4-tool \
Partial migration of config code generation to C. Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c. This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges. The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
2021-07-19 01:02:01 +02:00
zstd libzstd-dev bzip2 libbz2-dev libyaml-dev
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Some unit tests and all the integration tests require Docker. Running in containers allows us to simulate multiple hosts, test on different distributions and versions of PostgreSQL, and use sudo without affecting the host system.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
pgbackrest-dev => Install Docker
```
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker `whoami`
```
This clone of the pgBackRest repository is sufficient for experimentation. For development, create a fork and clone that instead.
pgbackrest-dev => Clone pgBackRest repository
```
git clone https://github.com/pgbackrest/pgbackrest.git
```
Use Rocky Linux for documentation builds instead of CentOS. Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time. Also update CentOS references to RHEL since that is what we are emulating for testing purposes.
2021-10-28 21:15:49 +02:00
If using a RHEL-based system, the CPAN XML parser is required to run `test.pl` and `doc.pl`. Instructions for installing Docker and the XML parser can be found in the `README.md` file of the pgBackRest [doc](https://github.com/pgbackrest/pgbackrest/blob/main/doc) directory in the section "The following is a sample RHEL 7 configuration that can be used for building the documentation". NOTE that the "Install latex (for building PDF)" section is not required since testing of the docs need only be run for HTML output.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
## Coding
The following sections provide information on some important concepts needed for coding within pgBackRest.
### Memory Contexts
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Memory is allocated inside contexts and can be long lasting (for objects) or temporary (for functions). In general, use `OBJ_NEW_BEGIN(MyObj)` for objects and `MEM_CONTEXT_TEMP_BEGIN()` for functions. See [memContext.h](https://github.com/pgbackrest/pgbackrest/blob/main/src/common/memContext.h) for more details and the [Coding Example](#coding-example) below.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Logging
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Logging is used for debugging with the built-in macros `FUNCTION_LOG_*()` and `FUNCTION_TEST_*()` which are used to trace parameters passed to/returned from functions. `FUNCTION_LOG_*()` macros are used for production logging whereas `FUNCTION_TEST_*()` macros will be compiled out of production code. For functions where no parameter is valuable enough to justify the cost of debugging in production, use `FUNCTION_TEST_BEGIN()/FUNCTION_TEST_END()`, else use `FUNCTION_LOG_BEGIN(someLogLevel)/FUNCTION_LOG_END()`. See [debug.h](https://github.com/pgbackrest/pgbackrest/blob/main/src/common/debug.h) for more details and the [Coding Example](#coding-example) below.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Logging is also used for providing information to the user via the `LOG_*()` macros, such as `LOG_INFO("some informational message")` and `LOG_WARN_FMT("no prior backup exists, %s backup has been changed to full", strZ(cfgOptionDisplay(cfgOptType)))` and also via `THROW_*()` macros for throwing an error. See [log.h](https://github.com/pgbackrest/pgbackrest/blob/main/src/common/log.h) and [error.h](https://github.com/pgbackrest/pgbackrest/blob/main/src/common/error.h) for more details and the [Coding Example](#coding-example) below.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Coding Example
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
The example below is not structured like an actual implementation and is intended only to provide an understanding of some of the more common coding practices. The comments in the example are only here to explain the example and are not representative of the coding standards. Refer to the Coding Standards document ([CODING.md](https://github.com/pgbackrest/pgbackrest/blob/main/CODING.md)) and sections above for an introduction to the concepts provided here. For an actual implementation, see [db.h](https://github.com/pgbackrest/pgbackrest/blob/main/src/db/db.h) and [db.c](https://github.com/pgbackrest/pgbackrest/blob/main/src/db/db.c).
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Example: hypothetical basic object construction
```c
/*
* HEADER FILE - see db.h for a complete implementation example
*/
// Typedef the object declared in the C file
typedef struct MyObj MyObj;
// Constructor, and any functions in the header file, are all declared on one line
MyObj *myObjNew(unsigned int myData, const String *secretName);
// Declare the publicly accessible variables in a structure with Pub appended to the name
typedef struct MyObjPub // First letter upper case
{
unsigned int myData; // Contents of the myData variable
} MyObjPub;
// Declare getters and setters inline for the publicly visible variables
// Only setters require "Set" appended to the name
__attribute__((always_inline)) static inline unsigned int
myObjMyData(const MyObj *const this)
{
return THIS_PUB(MyObj)->myData; // Use the built-in THIS_PUB macro
}
// Destructor
__attribute__((always_inline)) static inline void
myObjFree(MyObj *const this)
{
objFree(this);
}
// TYPE and FORMAT macros for function logging
#define FUNCTION_LOG_MY_OBJ_TYPE \
MyObj *
#define FUNCTION_LOG_MY_OBJ_FORMAT(value, buffer, bufferSize) \
FUNCTION_LOG_STRING_OBJECT_FORMAT(value, myObjToLog, buffer, bufferSize)
/*
* C FILE - see db.c for a more complete and actual implementation example
*/
// Declare the object type
struct MyObj
{
MyObjPub pub; // Publicly accessible variables must be first and named "pub"
const String *name; // Pointer to lightweight string object - see string.h
};
// Object constructor, and any functions in the C file, have the return type and function signature on separate lines
MyObj *
myObjNew(unsigned int myData, const String *secretName)
{
FUNCTION_LOG_BEGIN(logLevelDebug); // Use FUNCTION_LOG_BEGIN with a log level for displaying in production
FUNCTION_LOG_PARAM(UINT, myData); // When log level is debug, myData variable will be logged
FUNCTION_TEST_PARAM(STRING, secretName); // FUNCTION_TEST_PARAM will not display secretName value in production logging
FUNCTION_LOG_END();
ASSERT(secretName != NULL || myData > 0); // Development-only assertions (will be compiled out of production code)
MyObj *this = NULL; // Declare the object in the parent memory context: it will live only as long as the parent
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 17:10:35 +02:00
OBJ_NEW_BEGIN(MyObj) // Create a long lasting memory context with the name of the object
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
{
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 17:10:35 +02:00
this = OBJ_NEW_ALLOC(); // Allocate the memory required by the object
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
*this = (MyObj) // Initialize the object
{
.pub =
{
.myData = myData, // Copy the simple data type to this object
},
.name = strDup(secretName), // Duplicate the String data type to the this object's memory context
};
}
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 17:10:35 +02:00
OBJ_NEW_END();
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
FUNCTION_LOG_RETURN(MyObj, this);
}
// Function using temporary memory context
String *
myObjDisplay(unsigned int myData)
{
FUNCTION_TEST_BEGIN(); // No parameters passed to this function will be logged in production
FUNCTION_TEST_PARAM(UINT, myData);
FUNCTION_TEST_END();
String *result = NULL; // Result is created in the caller's memory context (referred to as "prior context" below)
MEM_CONTEXT_TEMP_BEGIN() // Begin a new temporary context
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 23:36:43 +02:00
String *resultStr = strNewZ("Hello"); // Allocate a string in the temporary memory context
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
if (myData > 1)
resultStr = strCatZ(" World"); // Append a value to the string still in the temporary memory context
else
LOG_WARN("Am I not your World?"); // Log a warning to the user
MEM_CONTEXT_PRIOR_BEGIN() // Switch to the prior context so the string duplication is in the caller's context
{
result = strDup(resultStr); // Create a copy of the string in the caller's context
}
MEM_CONTEXT_PRIOR_END(); // Switch back to the temporary context
}
MEM_CONTEXT_TEMP_END(); // Free everything created inside this temporary memory context - i.e resultStr
FUNCTION_TEST_RETURN(STRING, result); // Return result but do not log the value in production
}
// Create the logging function for displaying important information from the object
String *
myObjToLog(const MyObj *this)
{
return strNewFmt(
"{name: %s, myData: %u}", this->name == NULL ? NULL_Z : strZ(this->name), myObjMyData(this));
}
```
## Testing
A list of all possible test combinations can be viewed by running:
```
pgbackrest/test/test.pl --dry-run
```
While some files are automatically generated during `make`, others are generated by running the test harness as follows:
```
pgbackrest/test/test.pl --gen-only
```
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Prior to any submission, the html version of the documentation should also be run and the output checked by viewing the generated html on the local file system under `pgbackrest/doc/output/html`. More details can be found in the pgBackRest [doc/README.md](https://github.com/pgbackrest/pgbackrest/blob/main/doc/README.md) file.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
```
pgbackrest/doc/doc.pl --out=html
```
> **NOTE:** `ERROR: [028]` regarding cache is invalid is OK; it just means there have been changes and the documentation will be built from scratch. In this case, be patient as the build could take 20 minutes or more depending on your system.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Running Tests
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Examples of test runs are provided in the following sections. There are several important options for running a test:
- `--dry-run` - without any other options, this will list all the available tests
- `--module` - identifies the module in which the test is located
- `--test` - the actual test set to be run
- `--run` - a number identifying the run within a test if testing a single run rather than the entire test
- `--vm-out` - displays the test output (helpful for monitoring the progress)
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
- `--vm` - identifies the pre-built container when using Docker, otherwise the setting should be `none`. See [test.yml](https://github.com/pgbackrest/pgbackrest/blob/main/.github/workflows/test.yml) for a list of valid vm codes noted by `param: test`.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
For more options, run the test or documentation engine with the `--help` option:
```
pgbackrest/test/test.pl --help
pgbackrest/doc/doc.pl --help
```
#### Without Docker
If Docker is not installed, then the available tests can be listed using `--vm=none`, and each test must then be run with `--vm=none`.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
pgbackrest-dev => List tests that don't require a container
```
pgbackrest/test/test.pl --vm=none --dry-run
--- output ---
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: test begin on x86_64 - log level info
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: configure build
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: builds required: bin
v2.38: Minor Bug Fixes and Improvements IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported. Bug Fixes: * Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.) * Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.) Features: * Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.) * Add archive-missing-retry option. (Reviewed by Stefan Fercot.) * Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.) Improvements: * Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.) * Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.) * Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.) * Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.) * Do not retry fatal errors. (Reviewed by Reid Thompson.) * Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.) * Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.) Documentation Bug Fixes: * Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.) * Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.) * Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.) Documentation Improvements: * Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.) * Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.) * Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
2022-03-06 18:30:59 +02:00
--> P00 INFO: 72 tests selected
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
v2.38: Minor Bug Fixes and Improvements IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported. Bug Fixes: * Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.) * Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.) Features: * Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.) * Add archive-missing-retry option. (Reviewed by Stefan Fercot.) * Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.) Improvements: * Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.) * Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.) * Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.) * Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.) * Do not retry fatal errors. (Reviewed by Reid Thompson.) * Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.) * Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.) Documentation Bug Fixes: * Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.) * Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.) * Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.) Documentation Improvements: * Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.) * Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.) * Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
2022-03-06 18:30:59 +02:00
P00 INFO: P1-T01/72 - vm=none, module=common, test=error
[filtered 69 lines of output]
P00 INFO: P1-T71/72 - vm=none, module=performance, test=type
P00 INFO: P1-T72/72 - vm=none, module=performance, test=storage
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
--> P00 INFO: DRY RUN COMPLETED SUCCESSFULLY
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
pgbackrest-dev => Run a test
```
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
pgbackrest/test/test.pl --vm=none --vm-out --module=common --test=wait
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
--- output ---
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: test begin on x86_64 - log level info
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: autogenerate configure
P00 INFO: autogenerated version in configure.ac script: no changes
P00 INFO: autogenerated configure script: no changes
P00 INFO: autogenerate code
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: cleanup old data
P00 INFO: builds required: none
P00 INFO: 1 test selected
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: P1-T1/1 - vm=none, module=common, test=wait
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
v2.32: Repository Commands Bug Fixes: * Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.) Features: * Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.) * Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.) * Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.) Improvements: * Improve archive-get performance. (Reviewed by Cynthia Shang.)
2021-02-08 16:08:16 +02:00
run 1 - waitNew(), waitMore, and waitFree()
v2.34: PostgreSQL 14 Support Bug Fixes: * Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.) * Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.) * Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.) * Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.) Features: * Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.) * Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.) * Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.) * Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.) Improvements: * Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.) * Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.) * Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.) Documentation Improvements: * Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.) * Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.) * Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 12:51:08 +02:00
L0018 expect AssertError: assertion 'waitTime <= 999999000' failed
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
v2.34: PostgreSQL 14 Support Bug Fixes: * Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.) * Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.) * Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.) * Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.) Features: * Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.) * Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.) * Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.) * Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.) Improvements: * Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.) * Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.) * Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.) Documentation Improvements: * Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.) * Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.) * Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 12:51:08 +02:00
run 1/1 ------------- L0021 0ms wait
L0025 new wait
L0026 check remaining time
L0027 check wait time
L0028 check sleep time
L0029 check sleep prev time
L0030 no wait more
L0033 new wait = 0.2 sec
L0034 check remaining time
L0035 check wait time
L0036 check sleep time
L0037 check sleep prev time
L0038 check begin time
L0044 lower range check
L0045 upper range check
L0047 free wait
L0052 new wait = 1.1 sec
L0053 check wait time
L0054 check sleep time
L0055 check sleep prev time
L0056 check begin time
L0062 lower range check
L0063 upper range check
L0065 free wait
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
TESTS COMPLETED SUCCESSFULLY
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: P1-T1/1 - vm=none, module=common, test=wait
v2.28: Azure Repository Storage Bug Fixes: * Fix restore --force acting like --force --delta. This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue. (Reviewed by Cynthia Shang.) Features: * Azure support for repository storage. (Reviewed by Cynthia Shang, Don Seiler.) * Add expire-auto option. This allows automatic expiration after a successful backup to be disabled. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.) Improvements: * Asynchronous S3 multipart upload. (Reviewed by Stephen Frost.) * Automatic retry for backup, restore, archive-get, and archive-push. (Reviewed by Cynthia Shang.) * Disable query parallelism in PostgreSQL sessions used for backup control. (Reviewed by Stefan Fercot.) * PostgreSQL 13 beta2 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. * Improve handling of invalid HTTP response status. (Reviewed by Cynthia Shang.) * Improve error when pg1-path option missing for archive-get command. (Reviewed by Cynthia Shang.) * Add hint when checksum delta is enabled after a timeline switch. (Reviewed by Matt Bunter, Cynthia Shang.) * Use PostgreSQL instead of postmaster where appropriate. (Reviewed by Cynthia Shang.) Documentation Bug Fixes: * Fix incorrect example for repo-retention-full-type option. (Reported by Höseyin Sönmez.) * Remove internal commands from HTML and man command references. (Reported by Cynthia Shang.) Documentation Improvements: * Update PostgreSQL versions used to build user guides. Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version. (Reviewed by Stephen Frost.) * Update FAQ for expiring a specific backup set. (Contributed by Cynthia Shang. Reviewed by David Steele.) * Update FAQ to clarify default PITR behavior. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2020-07-20 14:57:22 +02:00
P00 INFO: tested modules have full coverage
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: writing C coverage report
P00 INFO: TESTS COMPLETED SUCCESSFULLY
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
An entire module can be run by using only the `--module` option.
pgbackrest-dev => Run a module
```
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
pgbackrest/test/test.pl --vm=none --module=postgres
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
--- output ---
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: test begin on x86_64 - log level info
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: autogenerate configure
P00 INFO: autogenerated version in configure.ac script: no changes
P00 INFO: autogenerated configure script: no changes
P00 INFO: autogenerate code
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: cleanup old data
P00 INFO: builds required: none
P00 INFO: 2 tests selected
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: P1-T1/2 - vm=none, module=postgres, test=client
P00 INFO: P1-T2/2 - vm=none, module=postgres, test=interface
v2.28: Azure Repository Storage Bug Fixes: * Fix restore --force acting like --force --delta. This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue. (Reviewed by Cynthia Shang.) Features: * Azure support for repository storage. (Reviewed by Cynthia Shang, Don Seiler.) * Add expire-auto option. This allows automatic expiration after a successful backup to be disabled. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.) Improvements: * Asynchronous S3 multipart upload. (Reviewed by Stephen Frost.) * Automatic retry for backup, restore, archive-get, and archive-push. (Reviewed by Cynthia Shang.) * Disable query parallelism in PostgreSQL sessions used for backup control. (Reviewed by Stefan Fercot.) * PostgreSQL 13 beta2 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. * Improve handling of invalid HTTP response status. (Reviewed by Cynthia Shang.) * Improve error when pg1-path option missing for archive-get command. (Reviewed by Cynthia Shang.) * Add hint when checksum delta is enabled after a timeline switch. (Reviewed by Matt Bunter, Cynthia Shang.) * Use PostgreSQL instead of postmaster where appropriate. (Reviewed by Cynthia Shang.) Documentation Bug Fixes: * Fix incorrect example for repo-retention-full-type option. (Reported by Höseyin Sönmez.) * Remove internal commands from HTML and man command references. (Reported by Cynthia Shang.) Documentation Improvements: * Update PostgreSQL versions used to build user guides. Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version. (Reviewed by Stephen Frost.) * Update FAQ for expiring a specific backup set. (Contributed by Cynthia Shang. Reviewed by David Steele.) * Update FAQ to clarify default PITR behavior. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2020-07-20 14:57:22 +02:00
P00 INFO: tested modules have full coverage
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: writing C coverage report
P00 INFO: TESTS COMPLETED SUCCESSFULLY
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### With Docker
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
Update test container OS versions. Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 23:29:20 +02:00
Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. `u20` for Ubuntu 20.04.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
pgbackrest-dev => Build a VM
```
Update test container OS versions. Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 23:29:20 +02:00
pgbackrest/test/test.pl --vm-build --vm=u20
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
--- output ---
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: test begin on x86_64 - log level info
v2.39: Verify and File Bundling Bug Fixes: * Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.) * Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.) Features: * Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.) * Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.) * PostgreSQL 15 support. (Reviewed by Stefan Fercot.) * Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.) * Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.) * Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.) * Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.) Improvements: * Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.) * Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.) * Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.) * Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.) * Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.) * Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.) * Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.) * Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.) * Add user:group to lock permission error. (Reviewed by Reid Thompson.) Documentation Bug Fixes: * Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.) * Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.) * Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.) Documentation Improvements: * Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.) * Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.) * Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.) * Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 14:46:24 +02:00
P00 INFO: Using cached pgbackrest/test:u20-base-20220504A image (45b0905785dfd06c867b4067563171970c7581c5) ...
Update test container OS versions. Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 23:29:20 +02:00
P00 INFO: Building pgbackrest/test:u20-test image ...
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: Build Complete
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
> **NOTE:** to build all the vms, just omit the `--vm` option above.
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
pgbackrest-dev => Run a Specific Test Run
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
pgbackrest/test/test.pl --vm=u20 --module=mock --test=archive --run=2
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
--- output ---
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: test begin on x86_64 - log level info
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: autogenerate configure
P00 INFO: autogenerated version in configure.ac script: no changes
P00 INFO: autogenerated configure script: no changes
P00 INFO: autogenerate code
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: cleanup old data and containers
P00 INFO: builds required: bin, bin host
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: build bin for u20 (/home/vagrant/test/bin/u20)
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: bin dependencies have changed, rebuilding
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
P00 INFO: build bin for none (/home/vagrant/test/bin/none)
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
P00 INFO: bin dependencies have changed, rebuilding
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: 1 test selected
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
Update test container OS versions. Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 23:29:20 +02:00
P00 INFO: P1-T1/1 - vm=u20, module=mock, test=archive, run=2
Suppress timestamps in contributing documentation. Timestamps cause a lot of churn and make it hard to tell if important changes are working.
2020-03-22 16:17:03 +02:00
P00 INFO: no code modules had all tests run required for coverage
P00 INFO: TESTS COMPLETED SUCCESSFULLY
Add building a development environment to contributing documentation. This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
2019-11-07 20:11:01 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Writing a Unit Test
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
The goal of unit testing is to have 100 percent code coverage. Two files will usually be involved in this process:
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- **define.yaml** - defines the number of tests to be run for each module and test file. There is a comment at the top of the file that provides more information about this file.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- **src/module/somefileTest.c** - where "somefile" is the path and name of the test file where the unit tests are located for the code being updated (e.g. `src/module/command/expireTest.c`).
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### define.yaml
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Each module is separated by a line of asterisks (*) and each test within is separated by a line of dashes (-). In the example below, the module is `command` and the unit test is `check`. The number of calls to `testBegin()` in a unit test file will dictate the number following `total:`, in this case 4. Under `coverage:`, the list of files that will be tested.
```
# ********************************************************************************************************************************
- name: command
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
test:
# ----------------------------------------------------------------------------------------------------------------------------
- name: check
total: 4
containerReq: true
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
coverage:
- command/check/common
- command/check/check
```
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### somefileTest.c
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Unit test files are organized in the `test/src/module` directory with the same directory structure as the source code being tested. For example, if new code is added to src/**command/expire**.c then test/src/module/**command/expire**Test.c will need to be updated.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Assuming that a test file already exists, new unit tests will either go in a new `testBegin()` section or be added to an existing section. Each such section is a test run. The comment string passed to `testBegin()` should reflect the function(s) being tested in the test run. Tests within a run should use `TEST_TITLE()` with a comment string describing the test.
```
// *****************************************************************************************************************************
if (testBegin("expireBackup()"))
{
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
TEST_TITLE("manifest file removal");
```
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Setting up the command to be run
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
The [harnessConfig.h](https://github.com/pgbackrest/pgbackrest/blob/main/test/src/common/harnessConfig.h) describes a list of functions that should be used when configuration options are required for a command being tested. Options are set in a `StringList` which must be defined and passed to the `HRN_CFG_LOAD()` macro with the command. For example, the following will set up a test to run `pgbackrest --repo-path=test/test-0/repo info` command on multiple repositories, one of which is encrypted:
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
```
StringList *argList = strLstNew(); // Create an empty string list
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 19:28:28 +02:00
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo"); // Add the --repo-path option
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 2, TEST_PATH "/repo2"); // Add the --repo2-path option
hrnCfgArgKeyRawStrId(argList, cfgOptRepoCipherType, 2, cipherTypeAes256Cbc); // Add the --repo2-cipher-type option
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, TEST_CIPHER_PASS); // Set environment variable for the --repo2-cipher-pass option
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
HRN_CFG_LOAD(cfgCmdInfo, argList); // Load the command and option list into the test harness
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
```
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Storing a file
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Sometimes it is desirable to store or manipulate files before or during a test and then confirm the contents. The [harnessStorage.h](https://github.com/pgbackrest/pgbackrest/blob/main/test/src/common/harnessStorage.h) file contains macros (e.g. `HRN_STORAGE_PUT` and `TEST_STORAGE_GET`) for doing this. In addition, `HRN_INFO_PUT` is convenient for writing out info files (archive.info, backup.info, backup.manifest) since it will automatically add header and checksum information.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Testing results
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Tests are run and results confirmed via macros that are described in [harnessTest.h](https://github.com/pgbackrest/pgbackrest/blob/main/test/src/common/harnessTest.h). With the exception of TEST_ERROR, the third parameter is a short description of the test. Some of the more common macros are:
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_RESULT_STR` - Test the actual value of the string returned by the function.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_RESULT_UINT` / `TEST_RESULT_INT` - Test for an unsigned integer / integer.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_RESULT_BOOL` - Test a boolean value.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_RESULT_PTR` / `TEST_RESULT_PTR_NE` - Test a pointer: useful for testing if the pointer is `NULL` or not equal (`NE`) to `NULL`.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_RESULT_VOID` - The function being tested returns a `void`. This is then usually followed by tests that ensure other actions occurred (e.g. a file was written to disk).
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `TEST_ERROR` / `TEST_ERROR_FMT` - Test that a specific error code was raised with specific wording.
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
> **NOTE:** `HRN_*` macros should be used only for test setup and cleanup. `TEST_*` macros must be used for testing results.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Testing a log message
If a function being tested logs something with `LOG_WARN`, `LOG_INFO` or other `LOG_*()` macro, then the logged message must be cleared before the end of the test by using the `TEST_RESULT_LOG()/TEST_RESULT_LOG_FMT()` macros.
```
TEST_RESULT_LOG(
"P00 WARN: WAL segment '000000010000000100000001' was not pushed due to error [25] and was manually skipped: error");
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
In the above, `Pxx` indicates the process (P) and the process number (xx), e.g. P00, P01.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Testing using child process
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
Sometimes it is useful to use a child process for testing. Below is a simple example. See [harnessFork.h](https://github.com/pgbackrest/pgbackrest/blob/main/test/src/common/harnessFork.h) for more details.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 17:58:23 +02:00
HRN_FORK_BEGIN()
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
{
Improve HRN_FORK*() macros. Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults. Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 20:22:53 +02:00
HRN_FORK_CHILD_BEGIN()
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
{
TEST_RESULT_INT_NE(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 23:36:43 +02:00
lockAcquire(cfgOptionStr(cfgOptLockPath), STRDEF("stanza1"), STRDEF("999-ffffffff"), lockTypeBackup, 0, true),
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
-1, "create backup/expire lock");
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
// Notify parent that lock has been acquired
HRN_FORK_CHILD_NOTIFY_PUT();
// Wait for parent to allow release lock
HRN_FORK_CHILD_NOTIFY_GET();
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
lockRelease(true);
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 17:58:23 +02:00
HRN_FORK_CHILD_END();
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 17:58:23 +02:00
HRN_FORK_PARENT_BEGIN()
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
{
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
// Wait for child to acquire lock
HRN_FORK_PARENT_NOTIFY_GET(0);
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 15:03:44 +02:00
HRN_CFG_LOAD(cfgCmdInfo, argListText);
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
TEST_RESULT_STR_Z(
infoRender(),
"stanza: stanza1\n"
" status: error (no valid backups, backup/expire running)\n"
" cipher: none\n"
"\n"
" db (current)\n"
" wal archive min/max (9.4): none present\n",
"text - single stanza, no valid backups, backup/expire lock detected");
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
// Notify child to release lock
HRN_FORK_PARENT_NOTIFY_PUT(0);
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 17:58:23 +02:00
HRN_FORK_PARENT_END();
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 17:58:23 +02:00
HRN_FORK_END();
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
#### Testing using a shim
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
A PostgreSQL libpq shim is provided to simulate interactions with PostgreSQL. Below is a simple example. See [harnessPq.h](https://github.com/pgbackrest/pgbackrest/blob/main/test/src/common/harnessPq.h) for more details.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
// Set up two standbys but no primary
harnessPqScriptSet((HarnessPq [])
{
HRNPQ_MACRO_OPEN_GE_92(1, "dbname='postgres' port=5432", PG_VERSION_92, "/pgdata", true, NULL, NULL),
HRNPQ_MACRO_OPEN_GE_92(8, "dbname='postgres' port=5433", PG_VERSION_92, "/pgdata", true, NULL, NULL),
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
// Close the "inner" session first (8) then the outer (1)
HRNPQ_MACRO_CLOSE(8),
HRNPQ_MACRO_CLOSE(1),
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
HRNPQ_MACRO_DONE()
});
TEST_ERROR(cmdCheck(), ConfigError, "primary database not found\nHINT: check indexed pg-path/pg-host configurations");
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Running a Unit Test
**Code Coverage**
Unit tests are run for all files that are listed in `define.yaml` and a coverage report generated for each file listed under the tag `coverage:`. Note that some files are listed in multiple `coverage:` sections for a module; in this case, each test for the file being modified should be specified for the module in which the file exists (e.g. `--module=storage --test=posix --test=gcs`, etc.) or, alternatively, simply run the module without the `--test` option. It is recommended that a `--vm` be specified since running the same test for multiple vms is unnecessary for coverage. The following example would run the test set from the **define.yaml** section detailed above.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u20
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
> **NOTE:** Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.
Because a test run has not been specified, a coverage report will be generated and written to the local file system under the pgBackRest directory `test/result/coverage/lcov/index.html` and a file with only the highlighted code that has not been covered will be written to `test/result/coverage/coverage.html`.
If 100 percent code coverage has not been achieved, an error message will be displayed, for example: `ERROR: [125]: c module command/check/check is not fully covered`
**Debugging with files**
Sometimes it is useful to look at files that were generated during the test. The default for running any test is that, at the start/end of the test, the test harness will clean up all files and directories created. To override this behavior, a single test run must be specified and the option `--no-cleanup` provided. Again, continuing with the check command, from **define.yaml** above, there are four tests. Below, test one will be run and nothing will be cleaned up so that the files and directories in `test/test-0` can be inspected.
```
Remove test.pl --smart, --dev, and --dev-test options. --smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it. Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 18:01:10 +02:00
pgbackrest/test/test.pl --vm-out --module=command --test=check --run=1 --no-cleanup
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
### Understanding Test Output
The following is a small sample of a typical test output.
```
run 8 - expireTimeBasedBackup()
run 8/1 ------------- L2285 no current backups
000.002s L2298 empty backup.info
000.009s 000.007s L2300 no backups to expire
```
**run 8 - expireTimeBasedBackup()** - indicates the run number (8) within the module and the parameter provided to testBegin, e.g. `testBegin("expireTimeBasedBackup()")`
**run 8/1 ------------- L2285 no current backups** - this is the first test (1) in run 8 which is the `TEST_TITLE("no current backups");` at line number 2285.
**000.002s L2298 empty backup.info** - the first number, 000.002s, is the time in seconds that the test started from the beginning of the run. L2298 is the line number of the test and `empty backup.info` is the test comment.
**000.009s 000.007s L2300 no backups to expire** - again, 000.009s, is the time in seconds that the test started from the beginning of the run. The second number, 000.007s, is the run time of the **previous** test (i.e. `empty backup.info` test took 000.007 seconds to execute). L2300 is the line number of the test and `no backups to expire` is the test comment.
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
## Adding an Option
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Remove generated config.auto.c file. This file duplicated the command list that already exists in parse.auto.c. Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 20:40:30 +02:00
Options can be added to a command or multiple commands. Options can be configuration file only, command-line only or valid for both. Once an option is successfully added, the `config.auto.h` and `parse.auto.c` files will automatically be generated by the build system.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
To add an option, two files need be to be modified:
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `src/build/config/config.yaml`
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing.xml with rendering changes from def7d513. Also update help.xml path missed in f4e1babf.
2021-10-13 17:43:14 +02:00
- `src/build/help/help.xml`
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
These files are discussed in the following sections along with how to verify the `help` command output.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### config.yaml
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
There are detailed comment blocks above each section that explain the rules for defining commands and options. Regarding options, there are two types: 1) command line only, and 2) configuration file. With the exception of secrets, all configuration file options can be passed on the command line. To configure an option for the configuration file, the `section:` key must be present.
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
The `option:` section is broken into sub-sections by a simple comment divider (e.g. `# Repository options`) under which the options are organized alphabetically by option name. To better explain this section, two hypothetical examples will be discussed. For more details, see [config.yaml](https://github.com/pgbackrest/pgbackrest/blob/main/src/build/config/config.yaml).
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
#### EXAMPLE 1 hypothetical command line only option
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
```
set:
type: string
command:
backup:
depend:
option: stanza
required: false
restore:
default: latest
command-role:
main: {}
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
Note that `section:` is not present thereby making this a command-line only option defined as follows:
- `set` - the name of the option
- `type` - the type of the option. Valid values for types are: `boolean`, `hash`, `integer`, `list`, `path`, `size`, `string`, and `time`
- `command` - list each command for which the option is valid. If a command is not listed, then the option is not valid for the command and an error will be thrown if it is attempted to be used for that command. In this case the valid commands are `backup` and `restore`.
- `backup` - details the requirements for the `--set` option for the `backup` command. It is dependent on the option `--stanza`, meaning it is only allowed to be specified for the `backup` command if the `--stanza` option has been specified. And `required: false` indicates that the `--set` option is never required, even with the dependency.
- `restore` - details the requirements for the `--set` option for the `restore` command. Since `required:` is omitted, it is not required to be set by the user but it is required by the command and will default to `latest` if it has not been specified by the user.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `command-role` - defines the processes for which the option is valid. `main` indicates the option will be used by the main process and not be passed on to other local/remote processes.
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
#### EXAMPLE 2 hypothetical configuration file option
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
repo-test-type:
section: global
type: string
group: repo
default: full
allow-list:
- full
- diff
- incr
command:
backup: {}
restore: {}
command-role:
main: {}
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `repo-test-type` - the name of the option
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
- `section` - the section of the configuration file where this option is valid (omitted for command line only options, see [Example 1](#example-1-hypothetical-command-line-only-option) above)
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `type` - the type of the option. Valid values for types are: `boolean`, `hash`, `integer`, `list`, `path`, `size`, `string`, and `time`
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `group` - indicates that this option is part of the `repo` group of indexed options and therefore will follow the indexing rules e.g. `repo1-test-type`.
Fix links and update child process example. Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 22:56:06 +02:00
- `default` - sets a default for the option if the option is not provided when the command is run. The default can be global (as it is here) or it can be specified for a specific command in the command section (as in [Example 1](#example-1-hypothetical-command-line-only-option) above).
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `allow-list` - lists the allowable values for the option for all commands for which the option is valid.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
- `command` - list each command for which the option is valid. If a command is not listed, then the option is not valid for the command and an error will be thrown if it is attempted to be used for that command. In this case the valid commands are `backup` and `restore`.
- `command-role` - defines the processes for which the option is valid. `main` indicates the option will be used by the main process and not be passed on to other local/remote processes.
At compile time, the `config.auto.h` file will be generated to contain the constants used for options in the code. For the C enums, any dashes in the option name will be removed, camel-cased and prefixed with `cfgOpt`, e.g. `repo-path` becomes `cfgOptRepoPath`.
Migrate command-line help generation to C. Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help. Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary. The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required. Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
2021-09-09 00:16:06 +02:00
### help.xml
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
All options must be documented or the system will error during the build. To add an option, find the command section identified by `command id="COMMAND"` section where `COMMAND` is the name of the command (e.g. `expire`) or, if the option is used by more than one command and the definition for the option is the same for all of the commands, the `operation-general title="General Options"` section.
To add an option, add the following to the `<option-list>` section; if it does not exist, then wrap the following in `<option-list>` `</option-list>`. This example uses the boolean option `force` of the `restore` command. Simply replace that with your new option and the appropriate `summary`, `text` and `example`.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
<option id="force" name="Force">
<summary>Force a restore.</summary>
<text>By itself this option forces the <postgres/> data and tablespace paths to be completely overwritten. In combination with <br-option>--delta</br-option> a timestamp/size delta will be performed instead of using checksums.</text>
<example>y</example>
</option>
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
> **IMPORTANT:** A period (.) is required to end the `summary` section.
### Testing the help
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
It is important to run the `help` command unit test after adding an option in case a change is required:
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
pgbackrest/test/test.pl --module=command --test=help --vm-out
```
To verify the `help` command output, build the pgBackRest executable:
```
pgbackrest/test/test.pl --vm=none --build-only
```
Use the pgBackRest executable to test the help output:
```
test/bin/none/pgbackrest help backup repo-type
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
### Testing the documentation
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
To quickly view the HTML documentation, the `--no-exe` option can be passed to the documentation generator in order to bypass executing the code elements:
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Fix typo in contributing guide. Not sure how this got broken but it was probably an errant search and replace.
2021-10-28 19:28:49 +02:00
pgbackrest/doc/doc.pl --out=html --no-exe
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
The generated HTML files will be placed in the `doc/output/html` directory where they can be viewed locally in a browser.
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
If Docker is installed, it will be used by the documentation generator to execute the code elements while building the documentation, therefore, the `--no-exe` should be omitted, (i.e. `pgbackrest/doc/doc.pl --output=html`). `--no-cache` may be used to force a full build even when no code elements have changed since the last build. `--pre` will reuse the container definitions from the prior build and saves time during development.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
The containers created for documentation builds can be useful for manually testing or trying out new code or features. The following demonstrates building through just the `quickstart` section of the `user-guide` without encryption.
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
pgbackrest/doc/doc.pl --out=html --include=user-guide --require=/quickstart --var=encrypt=n --no-cache --pre
Document how to contribute to pgBackRest. There's a lot more to be done here, but this is a good start.
2019-10-08 21:27:17 +02:00
```
Update contributing documentation. Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 19:21:25 +02:00
The resulting Docker containers can be listed with `docker ps` and the container can be entered with `docker exec doc-pg-primary bash`. Additionally, the `-u` option can be added for entering the container as a specific user (e.g. `postgres`).
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
## Submitting a Pull Request
Before submitting a Pull Request:
Update documentation/links to main branch.
2021-10-13 18:01:53 +02:00
- Does it meet the [coding standards](https://github.com/pgbackrest/pgbackrest/blob/main/CODING.md)?
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
- Have [Unit Tests](#writing-a-unit-test) been written and [run](#running-a-unit-test) with 100% coverage?
- If your submission includes changes to the help or online documentation, have the [help](#testing-the-help) and [documentation](#testing-the-documentation) tests been run?
- Has it passed continuous integration testing? Simply renaming your branch with the appendix `-cig` and pushing it to your GitHub account will initiate GitHub Actions to run CI tests.
When submitting a Pull Request:
- Provide a short submission title.
- Write a detailed comment to describe the purpose of your submission and any issue(s), if any, it is resolving; a link to the GitHub issue is also helpful.
Document required base branch for pull requests. Be explicit when submitting a PR about which branch to use as the base.
2022-05-10 00:07:11 +02:00
- Select the `integration` branch as the base for your PR, do not select `main` nor any other branch.
Update contributing documentation and add pull request template.
2021-07-21 19:14:06 +02:00
After submitting a Pull Request:
- One or more reviewers will be assigned.
- Respond to any issues (conversations) in GitHub but do not resolve the conversation; the reviewer is responsible for ensuring the issue raised has been resolved and marking the conversation resolved. It is helpful to supply the commit in your reply if one was submitted to fix the issue.
Lastly, thank you for contributing to pgBackRest!
Reference in New Issue Copy Permalink