1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1814 Commits

Author SHA1 Message Date
David Steele
aced5d47ed Replace cfgOptionGroupIdxToKey() with cfgOptionGroupName().
Do the replacement anywhere cfgOptionGroupIdxToKey() is being used to construct a group name in a message. cfgOptionGroupName() is better for this case since it also includes the name of the group so that it does not need to be repeated in each message.
2022-01-10 09:10:06 -05:00
David Steele
9657f1b325 Dynamically allocate index to key index map.
Now the config/config module has no notion of max option indexes. The config/parse still needs the max but this will be removed at a later date.
2022-01-09 19:53:05 -05:00
David Steele
e4b48eb430 Fix inconsistent group display names in messages.
In other instances there are no dashes, e.g. repo1 or pg1. Make these messages match.
2022-01-09 19:43:44 -05:00
Stefan Fercot
d866dd5c29
Add backup LSNs to info command output.
The backup LSNs are useful for performing LSN-based PITR. LSNs will not be displayed in the general text output (without --set) because they are probably not useful enough to deserve their own line.
2022-01-07 14:09:58 -05:00
David Steele
bb4b30ddd3
Remove support for PostgreSQL 8.3/8.4.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.

It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
2022-01-06 15:34:04 -05:00
Reid Thompson
a82f0179cd
Note that replications slots are not restored.
Update documentation and help to note that replication slots are not restored and reference the PostgreSQL documentation to explain why.
2022-01-04 16:11:27 -05:00
David Steele
75e4580919 Begin v2.38 development. 2022-01-03 08:52:25 -05:00
David Steele
f18f2d9991 v2.37: TLS Server
Bug Fixes:

* Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)
* Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)

Features:

* Add TLS server. (Reviewed by Stephen Frost, Reid Thompson, Andrew L'Ecuyer.)
* Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)

Improvements:

* Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)
* Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)
* Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)
* Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)
* Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)
* Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)

Documentation Improvements:

* Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)
* Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)
* Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)
2022-01-03 08:43:55 -05:00
David Steele
fccb7f7dd4 Add release note regarding IANA approval of the default TLS port. 2021-12-28 17:39:22 -05:00
Reid Thompson
6a12458440
Parse protocol/port in S3/Azure endpoints.
Utilize httpUrlNewParseP() to parse endpoint and port from the URL in the S3 and Azure helpers to avoid issues where protocol was not expected to be part of the URL.
2021-12-16 10:30:59 -05:00
David Steele
f06101de77 Add TLS server documentation.
Add documentation and make the feature visible.
2021-12-16 09:47:04 -05:00
David Steele
615bdff403
Fix socket leak on connection retries.
This leak was caused by the file descriptor variable getting clobbered after a long jump. Mark it as volatile to fix.

Testing this is a bit complex because the issue only happens in optimized builds, if at all. Put the test into the performance suite, which is always optimized, until a better idea presents itself.
2021-12-14 14:53:41 -05:00
David Steele
a73fe4eb96
Fix restore delta link mapping when path/file already exists.
If a path/file was remapped to a link using either --link-map or --link-all there would be no affect if the path/file already existed. If a link existed it would be properly updated and converting a link to a path/file also worked.

The issue happened during delta cleanup, which failed to check if the existing path/file had been remapped to a link.

Add checks for newly mapped path/file links and remove the old path/file we required.
2021-12-10 15:53:40 -05:00
Christoph Berg
c38e2d3170 Add verb to HTTP error output.
This makes it easier to debug HTTP errors.
2021-12-08 15:00:19 -05:00
David Steele
be4ac3923c
Error when restore is unable to find a backup to match the time target.
This was previously a warning but the warning is easy to miss so a lot of time may be lost restoring and recovering a backup that will not hit the target.

Since this is technically a breaking change, add an "important note" about the change to the release.
2021-12-08 13:57:26 -05:00
Stefan Fercot
6723305937
Add warning when checkpoint_timeout exceeds db-timeout.
In the backup command, add a warning if start-fast is disabled and the PostgreSQL checkpoint_timeout is greater than db-timeout.

In such cases, we might timeout before the checkpoint occurs and the backup really starts.
2021-12-08 12:29:20 -05:00
David Steele
bd2ba802db
Check that clusters are alive and correctly configured during a backup.
Fail the backup if a cluster stops or the standby is promoted. Previously, shutting down the primary would cause an error but it was not detected until the end of the backup. Now the error will happen sooner and a promotion on the standby will also cause an error.
2021-12-08 10:16:41 -05:00
David Steele
7b3ea883c7
Add SIGTERM and SIGHUP handling to TLS server.
SIGHUP allows the configuration to be reloaded. Note that the configuration will not be updated in child processes that have already started.

SIGTERM terminates the server process gracefully and sends SIGTERM to all child processes. This also gives the tests an easy way to stop the server.
2021-12-07 18:18:43 -05:00
David Steele
49145d72ba
Add timeline and checkpoint checks to backup.
Add the following checks:

* Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
* Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
* Standby is on the same timeline as the primary. If not, this standby is not following the primary.
* Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.
2021-12-07 09:21:07 -05:00
Reid Thompson
dcb4f09d83 Revert changes to backupFilePut() made in 1e77fc3d.
These changes were made obsolete by a3d7a23a.
2021-11-23 09:37:12 -05:00
Reid Thompson
a3d7a23a9d
Use infoBackupDataByLabel() to log backup size.
Eliminate summing and passing of copied files sizes for logging backup size.

Instead, utilize infoBackupDataByLabel() to pull the backup size for the log message.
2021-11-22 12:52:37 -05:00
Reid Thompson
1a0560d363
Allow y/n arguments for boolean command-line options.
This allows boolean boolean command-line options to work like their config file equivalents.

At least for now this behavior will remain undocumented since all examples in the documentation will continue to use the standard syntax. The idea is that it will "just work" when options are copied out of config files rather than generating an error.
2021-11-19 12:22:09 -05:00
David Steele
2d963ce947 Rename server-start command to server. 2021-11-18 17:23:11 -05:00
David Steele
1f14f45dfb
Check archive immediately after backup start.
Previously the archive was only checked at the end of the backup to ensure all WAL required to make the backup consistent was present. The problem was that if archiving was not functioning then the backup had to complete before the user found out, which could be a while if the database was large enough.

Add an archive check immediately after backup start so failures are reported earlier.

The trick is to determine which WAL to check. If the repo is new there may not be any WAL in it and pg_start_backup() will not switch the WAL segment if it is empty. These are both likely scenarios when setting up and/or testing pgBackRest.

If the WAL segment is switched by pg_start_backup(), then check the archive for the segment that was detected prior to backup start. This should be common on normal running clusters with regular activity. Note that this might not be the segment immediately prior to the backup start segment if WAL volume is high.

If pg_start_backup() did not switch the WAL then we can force a switch on PostgreSQL >= 9.3 by creating a restore point. In that case the WAL to check will be the backup start WAL. This is most likely to happen on idle systems, during testing, or immediately after a repo switch.

An advantage of this approach other than earlier notification is that the backup directory will not be created so no resume will be attempted on the next backup.

Note that some additional churn was created in backup.c because the load of archive.info needs to be done earlier.
2021-11-18 16:18:10 -05:00
David Steele
dea752477a Remove obsolete statement about future multi-repository support. 2021-11-17 16:39:04 -05:00
Reid Thompson
1e77fc3d75
Include backup_label and tablespace_map file sizes in log output.
In cases where they are returned by postgres, include backup_label and tablespace_map file sizes in the backup size value output in the log.
2021-11-16 10:21:32 -05:00
David Steele
6b5322cdad Add findutils package to RHEL 8 documentation container.
This package was dropped from the most recent Rocky Linux 8 image.
2021-11-16 09:27:15 -05:00
David Steele
df89eff429 Fix typos and improve documentation for the tablespace-map-all option. 2021-11-15 16:53:41 -05:00
David Steele
e62ba8e85e Add path to pgbench used for stress test in user guide.
This allows the stress test to run on RHEL.
2021-11-12 17:15:45 -05:00
David Steele
afe77e76e0 Update contributor for 6e635764. 2021-11-10 07:31:02 -05:00
Reid Thompson
6e635764a6
Match backup log size with size reported by info command.
Properly log the size of files copied during the backup, matching the backup size returned from the info command.

In the reference issue, the incremental backup after switchover logs the size of all files evaluated rather than only the size of the files copied in the backup.
2021-11-09 13:24:56 -05:00
David Steele
038abaa71d
Display size option default and allowed values with appropriate units.
Size option default and allowed values were displayed in bytes, which was confusing for the user.

This also lays the groundwork for adding units to time options.

Move option parsing functions into a common module so they can be used from the build module.
2021-11-03 15:23:08 -04:00
Reid Thompson
2a576477b3
Add --cmd option.
Allows users to provide an executable to be used when pgbackrest generates command strings that expect to invoke pgbackrest. These generated commands are written to files by pgbackrest, e.g. recovery.conf.
2021-11-03 11:36:34 -04:00
David Steele
c5b5b58806
Simplify error handler.
The error handler used a loop to process try, catch, and finally blocks. This worked fine but static analysis tools like Coverity did not understand that the finally block would always run and so there were false positives about double-free, unfreed resource, etc.

This implementation removes the loop, which simplifies everything, and makes it clear that the finally block will always run. This cuts down on Coverity false positives.

This implementation also catches lack of coverage on empty catch blocks so a few test fixes were committed separately in d74fe7a.

A small refactor in backup.c is required because gcc 10.3.1 on Fedora 33 complains that the reason variable may be used uninitialized. It's not clear why this is the case, but reducing the scope of the TRY block fixes the issue.
2021-11-03 10:36:31 -04:00
David Steele
7f6c513be9
Add StringId as an option type.
Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
David Steele
b13844086d
Use cfgOptionStrId() instead of cfgOptionStr() where appropriate.
The compress-type, repo-type and log-level-* options have allow lists, which means it is more efficient to treat them as StringIds.

For compress-type and log-level-* also update the functions that convert them to enums.
2021-11-01 17:35:19 -04:00
David Steele
bc352fa6a8
Simplify strIdFrom*() functions.
The strIdFrom*() forced the caller to pick an encoding, which led to a number of TRY...CATCH blocks in the code. In practice the caller does not care which encoding is used as long as the string is valid for some encoding.

Update the strIdFrom*() function to try all possible encodings and only throw an error when the string is not valid for any of them.
2021-11-01 10:08:56 -04:00
David Steele
904b897f5e Begin v2.37 development. 2021-11-01 09:03:42 -04:00
David Steele
42fd6ce4e0 v2.36: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow "global" as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)
* Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)

Improvements:

* Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)
* Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)
* Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Benjamin Blattberg, Andrew L'Ecuyer.)
* Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)
* Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)
* Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)
* Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)
2021-11-01 08:59:14 -04:00
David Steele
1336657326 Restore some linefeed rendering behavior from before def7d513.
The new rendering behavior is correct in normal cases, but for the pre-rendered HTML blocks in the command and configuration references it causes a lot of churn. This would be OK if the new HTML was diff-able, but it is not.

Go back to the old behavior of using br tags for this case to reduce churn until a more permanent solution is found.
2021-10-29 10:35:56 -04:00
David Steele
c32e000ab9 Use Rocky Linux for documentation builds instead of CentOS.
Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time.

Also update CentOS references to RHEL since that is what we are emulating for testing purposes.
2021-10-28 15:15:49 -04:00
David Steele
4f10441574 Add missing paragraph tags in coding standards. 2021-10-26 08:25:21 -04:00
David Steele
7fb99c59c8 Use externed instead of extern'd in comments.
This is mostly to revert some comment changes in b11ab9f7 that will break the ppc64le patch, but at the same time keep the spelling consistent in all comments and documentation.

Also revert some space changes for the same reason.
2021-10-26 07:46:48 -04:00
David Steele
653ffcf8d9 Adjustments for new breaking change in Azurite.
Azurite released another breaking change (see fbd018cd, 096829b3, c38d6926, and Azurite issue 1039) so make adjustments as needed to documentation and tests.

Also remove some dead code that hid the repo-storage-host option and was made obsolete by all these changes.
2021-10-25 15:42:28 -04:00
David Steele
13d4559708 Check return value of getsockopt().
Checking the return value is not terribly important here, but if setsockopt() fails it is likely that bind() will fail as well. May as well get it over with and this makes Coverity happy.
2021-10-25 15:31:39 -04:00
Reid Thompson
1152f7a7d6 Fix mismatched parameters in tlsClientNew() call.
3879bc69 added this call and the parameters were not quite right but in way that the compiler decided they were OK. It was mostly working but TLS verification was disabled if caPath was NULL, which is not OK.
2021-10-25 12:56:33 -04:00
David Steele
3879bc69b8
Add WebIdentity authentication for AWS S3.
This allows credentials to be automatically acquired in an EKS environment.
2021-10-22 18:31:55 -04:00
David Steele
51785739f4
Store config values as a union instead of a variant.
The variants were needed to easily serialize configurations for the Perl code.

Unions are more efficient and will allow us to add new types that are not supported by variants, e.g. StringId.
2021-10-22 18:02:20 -04:00
David Steele
2cea005f74 Fix segfault on invalid GCS key file. 2021-10-22 17:19:16 -04:00
David Steele
a63e732987 Fix indentation. 2021-10-21 17:25:32 -04:00
David Steele
78e1bd3330 Move v1 documentation links out of the introduction.
There should be few if any users running v1 now so these links do not need to be so prominent.
2021-10-21 17:10:00 -04:00
David Steele
861df2a73c Add GitHub repository link to index.html and README.md. 2021-10-21 17:02:46 -04:00
Reid Thompson
1cb8ae15de
Fix incorrect host name in user guide.
The text indicates to populate the pg-primary IP address into the pg_hba.conf file to allow replication connections. It should indicate to populate the pg-standby IP address
2021-10-21 13:51:59 -04:00
David Steele
fbd018cd56 Allow S3/Azure Docker images to be specified in user guide.
It is not uncommon for the S3/Azure emulators we use to introduce breaking changes without warning. If that happens the documentation can still be built by specifying a working version of the image. In general, it is better to let the version float so we know when things break.

Azurite has yet another breaking change coming up (see 096829b3, c38d6926, and Azurite issue 1039) so set azure-image at the current version until the breaking change has been released.
2021-10-20 08:22:37 -04:00
David Steele
e443e3c6c0 Add br tags for HTML documentation rendering missed in def7d513. 2021-10-19 09:06:06 -04:00
Reid Thompson
6cc8e45df6 Add missing paragraph tag in user guide. 2021-10-18 14:45:36 -04:00
David Steele
ccc255d3e0 Add TLS Server.
The TLS server is an alternative to using SSH for protocol connections to remote hosts.

This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
David Steele
b16e827d69 Do not show output of PostgreSQL upgrade status check in user guide.
On some platforms the output may contain UTF-8 characters that the latex code is not prepared to handle.

Showing the command is much more important than showing the output, so no big loss.
2021-10-13 13:20:11 -04:00
David Steele
430efff98a Update documentation/links to main branch. 2021-10-13 12:01:53 -04:00
David Steele
1212668d5e Update contributing.xml with rendering changes from def7d513.
Also update help.xml path missed in f4e1babf.
2021-10-13 11:43:14 -04:00
David Steele
498902e885
Allow "global" as a stanza prefix.
A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.

Update the config parser to correctly recognize global sections.
2021-10-07 12:18:24 -04:00
David Steele
338102861f Improve instructions for rebuilding pgbackrest during stress testing. 2021-10-07 11:01:48 -04:00
David Steele
68c5f3eaf1
Allow link-map option to create new links.
Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.

Allow link-map to create a new link as long as a valid path/file from the backup is referenced.
2021-10-05 17:59:05 -04:00
David Steele
6af827cbb1
Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
Stefan Fercot
34f7873432
Report backup file validation errors in backup.info.
Currently errors found during the backup are only available in text output when specifying --set.

Add a flag to backup.info that is available in both the text and json output when --set is not specified. This at least provides the basic info that an error was found in the cluster during the backup, though details are still only available as described above.
2021-10-04 13:45:53 -04:00
David Steele
71047a9d6d Use strncpy() to limit characters copied to optionName.
Valgrind complained about uninitialized values on arm64 when comparing the reset prefix, probably because "reset" ended up being larger than the option name: Conditional jump or move depends on uninitialised value(s) at cfgParseOption (parse.c:568).

Coverity complained because it could not verify the size of the string to be copied into optionName, probably because it does not understand the purpose of strSize(): You might overrun the 65-character fixed-size string "optionName" by copying the return value of "strZ" without checking the length.

Use strncpy() even though we have already checked the size and make sure the string is terminated. Keep the size check because searching for truncated option names is not a good idea.

This is not a production bug since the code has not been released yet.
2021-10-02 16:17:33 -04:00
David Steele
9e79f0e64b Add recovery start time to online backup restore log.
This helps give an idea of how much recovery needs to be done to reach the end of the WAL stream and is easier to read than the backup label.
2021-09-29 10:31:51 -04:00
David Steele
9346895f5b
Rename page checksum error to error list in info text output.
"error list" makes it clearer that other errors may be reported. For example, if checksum-page is true in the manifest but no checksum-page-error list is provided then the error is in alignment, i.e. the file size is not a multiple of the page size, with allowances made for a valid-looking partial page at the end of the file.

It is still not possible to differentiate between alignment and page checksum errors in the output but this will be addressed in a future commit.
2021-09-29 09:58:47 -04:00
David Steele
b7ef12a76f Add hints to standby replay timeout message. 2021-09-28 15:55:13 -04:00
David Steele
096829b3b2 Add repo-azure-uri-style option.
Azurite introduced a breaking change in 8f63964e to use automatically host-style URIs when the endpoint appears to be a multipart hostname.

This option allows the user to configure which style URI will be used, but changing the endpoint might cause breakage if Azurite decides to use a different style. Future changes to Azurite may also cause breakage.
2021-09-27 09:01:53 -04:00
David Steele
c38d6926d6 Revert Azurite version for testing to 3.14.0.
3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.
2021-09-09 08:48:45 -04:00
David Steele
f4e1babf6b Migrate command-line help generation to C.
Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.

Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.

The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.

Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
2021-09-08 18:16:06 -04:00
David Steele
def7d513cd Eliminate linefeed formatting from documentation.
Linefeeds were originally used in the place of <p> tags to denote a paragraph. While much of the linefeed usage has been replaced over time, there were many places where it was still being used, especially in reference.xml. This made it difficult to get consistent formatting across different output types. In particular there were formatting issues in the command-line help because it is harder to audit than HTML or PDF.

Replace linefeed formatting with proper <p> tags to make formatting more consistent.

Remove double spaces in all text where <p> tags were added since it does not add churn.

Update all <ul>/<ol>/<li> tags to the more general <list>/<list-item> tags.

Add a few missing periods.
2021-09-08 17:35:45 -04:00
David Steele
475b57c89b Allow additional memory to be allocated with a mem context.
The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.

Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.

Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.

Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
David Steele
02b06aa495
Increase max index allowed for pg/repo options to 256.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.

Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.

Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.

Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.

cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 12:09:50 -04:00
David Steele
aee0e7bac7 Begin v2.36 development. 2021-08-23 07:03:40 -04:00
David Steele
3787cf7803 v2.35: Binary Protocol
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.

Bug Fixes:

* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)

Improvements:

* Binary protocol. (Reviewed by Cynthia Shang.)
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)

Documentation Bug Fixes:

* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)

Documentation Improvements:

* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)

Test Suite Improvements:

* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-08-23 06:52:51 -04:00
David Steele
0354144c90 Add instructions for dev builds to stress test in user guide. 2021-08-19 11:35:49 -04:00
David Steele
56ad427653 Add variables for build paths in the user guide.
This eliminates repetition of the build path so it can be changed more easily.

Also create the build path explicitly rather than suggest that the user do it.
2021-08-18 10:52:35 -04:00
David Steele
944a9e8ff1 Reduce memory requirements of stress test section in the user guide.
The standby memory was set to 1024mb in 86a651f9 to compensate for a memory leak in restore. The leak has been fixed (or at least mitigated) in e1e6e475 and 4fb6384f so the memory can be reduced to 512mb, the same as the primary.
2021-08-18 10:17:10 -04:00
David Steele
4fb6384f10 Fix more memory leaks introduced by the binary protocol in 6a1c0337.
Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
Cynthia Shang
eca2fc6958
Update config/parse test to use standard patterns. 2021-08-12 12:38:07 -04:00
Cynthia Shang
e17865a03a
Update protocol/protocol test to use standard patterns. 2021-08-12 11:57:17 -04:00
David Steele
86a651f9b0 Add optional stress test section to the user guide.
This is intended to provide pre-release stress-testing. Include container memory limits to help check for memory leaks.

Also add parallelism to make for faster builds.
2021-08-12 07:49:59 -04:00
David Steele
a0bdfa436c
Log backup file total and restore size/file total.
The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.

Add size/file total log message to restore since it did not exist before.
2021-08-11 13:39:36 -04:00
David Steele
6ab18dc0fa
Rearrange backup documentation in user guide.
Remove the "Automatic Stop Option" section since it only applies to PostgreSQL <= 9.6, which will soon be EOL. Since we no longer build the user guide for PostgreSQL < 10 this section was no longer being tested. The stop-auto option is still documented in the reference.

Move the "Fast Start Option" to "Quick Start - Perform Backup". This is a commonly-used option so it makes sense to mention it earlier. This also makes the backups run more quickly. In the worst case, backups in "Quick Start - Perform Backup" could take minutes to start

Move the "Archive Timeout" section to "Quick Start - Perform Backup" since it is the last section in "Backup".
2021-08-11 12:59:25 -04:00
David Steele
f716cb6f4f Fix use after free introduced by the binary protocol in 6a1c0337.
The user and group were stored in a temp reset mem context so they could get freed if there were enough files to trigger the reset in storageRemoteInfoList().

Allocate user and group in a mem context provided by the caller to prevent them being freed prematurely.
2021-08-10 14:22:38 -04:00
Cynthia Shang
71b654fc29
Fix links and update child process example.
Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().
2021-08-09 16:56:06 -04:00
Cynthia Shang
f653b59664
Update db/db test to use standard patterns. 2021-08-09 16:35:48 -04:00
David Steele
2d5cbfc731 Fix memory leaks introduced by the binary protocol in 6a1c0337.
Additional buffers were being allocated for the protocol messages but not being freed.

Most of the allocations were fairly harness, but storageRemoteOpenReadProtocol() and storageWriteRemote() were problematic because they were allocating (but not freeing) buffers equal to the transfer size of the file. Depending on compression, this could be a lot of memory. Though the memory was freed after each file transfer the aggregate of memory used during parallel processing could overwhelm systems with constrained memory.

Also allocate larger initial buffers in storageRemoteOpenReadProtocol() and storageWriteRemote() so a reallocation is not needed.
2021-08-09 09:28:17 -04:00
David Steele
935de1e7a0
Only pass selected repo options to the remote.
Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.

Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
David Steele
155ef85013
Fix incorrect host names in user guide.
pg1 was incorrectly used instead of {[host-pg1]} which meant the wrong host name was displayed.

Also, the install block was installing packages to the build host no matter which host was specified.
2021-08-03 14:48:22 -04:00
David Steele
b47a07b8b9 Remove generated config.auto.c file.
This file duplicated the command list that already exists in parse.auto.c.

Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 14:40:30 -04:00
David Steele
e32f9e146b
Add check for test path inside repo path.
If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.

Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.
2021-07-29 11:09:51 -04:00
Cynthia Shang
eeaab6a3d1
Update config tests (except parse) to use standard patterns. 2021-07-23 16:18:50 -04:00
Cynthia Shang
4c9ddf5ef2
Update storage tests to use standard patterns.
The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.

Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.

This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c
2021-07-22 18:17:08 -04:00
Cynthia Shang
e3d05e2b4a
Update contributing documentation and add pull request template. 2021-07-21 13:14:06 -04:00
Cynthia Shang
4992508764
Update command/restore test to use standard patterns. 2021-07-20 13:30:17 -04:00
David Steele
9ee9b1fad6 Remove test.pl --smart, --dev, and --dev-test options.
--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.

Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 12:01:10 -04:00
Cynthia Shang
a6cdf5b22f
Update command/repo test to use standard patterns. 2021-07-19 10:47:49 -04:00
Cynthia Shang
ebe5848494
Update command/local and command/remote tests to use standard patterns.
Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.
2021-07-19 09:51:11 -04:00
David Steele
c5ae047e76 Partial migration of config code generation to C.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.

This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.

The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
2021-07-18 19:02:01 -04:00
David Steele
6397d73535 Update test container OS versions.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.

Add --no-install-recommends to apt-get commands to save a bit of time and space.

Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.

Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 17:29:20 -04:00
Cynthia Shang
117ffe8896
Update command/help tests to use standard patterns.
In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.

Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.
2021-07-15 17:51:48 -04:00
Cynthia Shang
4ad0bbda53
Update command/backup tests to use standard patterns.
Includes backup and backupCommon tests.

Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.

String values were also removed where they were no longer needed.
2021-07-15 17:00:20 -04:00
David Steele
25508a48e0 Add contributors missed in 8e187cdb. 2021-07-15 14:34:47 -04:00
Stefan Fercot
563d117967
Allow restore --type=lsn.
The LSN recovery target was added in PostgreSQL 10 but did not get added to pgBackRest.
2021-07-15 13:02:41 -04:00
Stefan Fercot
8fac1ac50d
Loop while waiting for checkpoint LSN to reach replay LSN.
It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.

Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.
2021-07-15 07:45:06 -04:00
Cynthia Shang
a6691c6f61 Clarify restore --type behavior in command reference. 2021-07-14 16:06:42 -04:00
David Steele
1ace1ac938 Improve HRN_FORK*() macros.
Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.

Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
David Steele
76cfbf833d Rename HARNESS_FORK*() macros to HRN_FORK*().
This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
Cynthia Shang
f2ec4e00a4 Add cipher parameters to TEST_STORAGE_GET() macro. 2021-07-09 15:59:17 -04:00
David Steele
3c8819e10f
Add CodeQL static code analysis.
Also fix some minor issues identified, specifically using gmtime_r()/localtime_r() vs gmtime()/localtime().
2021-07-09 14:16:10 -04:00
David Steele
849ab343aa Change level of backup/restore copied file logging to detail.
The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
2021-07-09 13:50:35 -04:00
Cynthia Shang
8bb0b28455
Update command/control and command/command tests to use standard patterns.
In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:

1. to ensure that an inadvertent typo in the path/file name does not go undetected,
2. to ensure that nothing else has removed the file prior to the call, and
3. consistency

Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.
2021-07-08 16:34:11 -04:00
David Steele
8e1807cdbe
Detect errors in S3 multi-part upload finalize.
Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.
2021-07-08 13:06:52 -04:00
David Steele
43b874628c
Fix detection of circular symlinks.
Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.

Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.
2021-07-08 11:30:23 -04:00
Stefan Fercot
056a3070e0
Automatically create data directory on restore.
Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.

Remove the check and update tests to create the data directory at least once.
2021-07-08 10:33:19 -04:00
Cynthia Shang
62e8d97af0
Update command/archive tests to use standard patterns.
Includes archiveCommon, archiveGet and archivePush.

Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Eric Radman
23bdc3deb6
Fix documentation and comment typos.
Identified using `ag -l | igor`.
2021-07-01 11:50:03 -04:00
Cynthia Shang
6a90d46909
Update info/manifest test to use standard patterns. 2021-06-30 08:51:11 -04:00
David Steele
39c1c10120 Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.
It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
Cynthia Shang
c6208113bf Add timeModified parameter to HRN_STORAGE_PUT() macro. 2021-06-28 12:14:26 -04:00
Cynthia Shang
065a8c8454
Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
Cynthia Shang
fb2d111dcd
Update info/infoBackup test to use standard patterns. 2021-06-24 16:11:14 -04:00
David Steele
6a1c0337dd
Binary protocol.
Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol.

There are a number advantages:

* The pack type is more compact than JSON and are more efficient to render/parse.
* Packs are more strictly typed than JSON.
* Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state.
* There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files.

Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C.

There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
Cynthia Shang
bffb43ea3f
Update command/check test to use standard patterns. 2021-06-24 12:23:09 -04:00
Cynthia Shang
17c9ed0ef0
Update command/info test to use standard patterns.
Some tests had to be reordered or updated, as follows:

* Reordered tests at line 317 and 331 to avoid unnecessary file removal.

* Change "stanza found" test at line 1735 to reflect real-life scenario. Originally this test had the cipher-pass environment key set up which caused the RepoGrp to be 2 but with no valid repo path. This resulted in the repo loops executing for the repo2 but since the path was not defined, the tests just reported "none" for cipher which is incorrect since the repo IS encrypted.

* Moved order of HRN_CFG_LOAD in some tests when able to avoid using storageTest.
2021-06-24 08:55:44 -04:00
Cynthia Shang
ff95eddc36
Update command/verify test to use standard patterns. 2021-06-23 18:09:46 -04:00
Cynthia Shang
8dd882bade
Update info/infoArchive test to use standard patterns. 2021-06-23 16:20:47 -04:00
David Steele
e697f5705a
Clear error when a CATCH() block finishes.
It is better to clear errors after the catch block completes rather than leave them set until the next error. This also make is possible to tell when a error is currently being handled, which a function further down the stack might use to modify its behavior. Currently this is only useful in testing, but clearing the error seems like a good idea in general.

Two places used errors outside the CATCH() block. Mem context cleanup now uses a FINALLY() which is a better implementation anyway. The error handling in main() now calls exitSafe() from withing the CATCH() block.
2021-06-23 13:02:19 -04:00
Cynthia Shang
03021c6a17 Update command/stanza test to use standard patterns.
No core code changes; only changes for stanza-create, stanza-upgrade, and stanza-delete command unit tests.
2021-06-14 13:28:27 -04:00
David Steele
4a075b7252
Add support for more Pack types.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).

Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.

Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
2021-06-08 12:55:00 -04:00
Cynthia Shang
3f9fbc3c24
Update command/expire test to use standard patterns.
The way tests are written has evolved over time. Update the command/expire test to use the new test patterns.
2021-06-08 11:00:28 -04:00
David Steele
a7b1d2b9d0 Begin v2.35 development. 2021-06-07 07:32:34 -04:00
David Steele
d10a99d73b v2.34: PostgreSQL 14 Support
Bug Fixes:

* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)

Features:

* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)

Improvements:

* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)

Documentation Improvements:

* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 06:51:08 -04:00
David Steele
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
Cynthia Shang
aceb956815
Change archive expiration logging from detail to info level.
To assist with issue debugging, removal of files during archive expiration has been changed from detail level logging to info level.
2021-05-27 08:03:44 -04:00
Stefan Fercot
5b332b22b5
Add repo-retention-history option to expire backup history.
The default is to keep all backup history to match the current behavior. In minimal configuration (0 days), unexpired backups are always kept in history.

When a full backup manifest expires, all dependent differential/incremental manifests expire as well.
2021-05-26 13:07:18 -04:00
David Steele
ba351e9c5c Refactor storage/remote unit test using the protocol remote shim.
Using the local process shim improves coverage and simplifies the tests.
2021-05-26 12:38:23 -04:00
David Steele
441c000b5c Factor remote process exec out of protocolRemoteGet().
This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.

The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
David Steele
6732806d89 Add local process shim to archive-get/archive-push unit tests.
Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
David Steele
d55b9471a8 Protocol shim improvements.
Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.

Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.

Log shim at debug level to make it obvious in the logs when a shim is in use.
2021-05-25 11:00:24 -04:00
David Steele
55f52955a5 Fix shims with more than one function.
Each subsequent function would double every line in the C module.

Move the loop to detect shimmed functions down to prevent this issues.
2021-05-24 17:45:31 -04:00
David Steele
2452c4d5a4
Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
David Steele
eba013b49b
Fix issue when checking links for large numbers of tablespaces.
manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.

Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.
2021-05-24 16:29:36 -04:00
David Steele
ccac75e7de
Consolidate RHEL/CentOS user guide into a single document.
The user guide was split primarily to provide documentation for the stop-auto option in PostgreSQL <= 9.5. Now that 9.5 is EOL there does not seem to be a good reason to generate an extra user guide. The stop-auto function is still documented in the reference.

Leave the stop-auto documentation in the user guide in case we want to manually generate documentation for older versions.

Also rename centos to rhel for most identifiers since that is the core platform we are building for, similar to how we label 'debian' builds even though we generally use Ubuntu. With CentOS set to become an upstream for RHEL later this year, we'll likely need to pick a new test distribution, perhaps Rocky Linux if that gets off the ground.
2021-05-24 16:03:15 -04:00
David Steele
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
Cynthia Shang
68faf1482a
Update contributing documentation.
Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 13:21:25 -04:00
David Steele
15b8b9207d Add log shim.
This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.
2021-05-21 12:51:32 -04:00