1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3699 Commits

Author SHA1 Message Date
David Steele
e62e6664d3 Remove release note about the default port being provisional.
The default port has now been approved by IANA.
2022-01-26 13:26:22 -06:00
David Steele
cf5b3a302f Fix language in rh7 test container for aarch64.
The /etc/profile.d/lang.sh script was causing issues but it does not exist on amd64, so it seems the easiest thing was to remove it.

Fix how 32-bit VMs are determined now that another 64-bit architecture has been added.

And remove some obsolete VM hashes.
2022-01-26 13:22:31 -06:00
David Steele
da0f3a8553 Download correct key on aarch64 in the user guide.
The default key that gets installed is for amd64. This is only required for RHEL8 it seems, so something is definitely amiss.
2022-01-26 10:15:14 -06:00
David Steele
e4df5b7d38
Simplify manifest file defaults.
Previously manifest load required two passes through the file list, one to load the data and one to set the defaults. This required each file to be packed twice.

Instead simply note that the file value is default and then set the file defaults when they are loaded from the manifest. This is made possible by the different internal/external representations for files so the same method cannot be applied to paths and links.

This change seems to resolve the performance issues noted in 61ce586 but there is no obvious reason why.
2022-01-24 15:21:07 -06:00
David Steele
044138fbb1 Centralize common PostgreSQL options in the user guide.
Centralize these options so they are consistent across clusters.

Also, there were some options that the user doesn't really need to see, .e.g. log_line_prefix. These can be set in advance so they don't need to be part of the documentation.
2022-01-23 15:07:13 -05:00
David Steele
13623d6ee5 Create create_test_table() function earlier in user guide.
This function (which creates lots of tables) is generally useful for testing (not just stress testing) so create it as soon as the cluster is created.

Also add the data parameter which will insert a single row into the table so the file on disk is not zero bytes.
2022-01-23 13:53:52 -05:00
David Steele
ca13f11b4a Optimization for jsonFromStrInternal().
This is an extremely hot code path when saving the manifest so every little bit helps.
2022-01-22 17:20:25 -05:00
David Steele
61ce58692f
Pack manifest file structs to save memory.
Manifests with a very large number of files can use a considerable amount of memory. There are a lot of zeroes in the data so it can be stored more efficiently by using base-128 varint encoding for the integers and storing the strings in the same allocation.

The downside is that the data needs to be unpacked in order to be used, but in most cases this seems fast enough (about 10% slower than before) except for saving the manifest, which is 10% slower up to 10 million files and then gets about 5x slower by 100 million (two minutes on my M1 Mac). Profiling does not show this slowdown so I wonder if this is related to the change in memory layout. Curiously, the function that increased most was jsonFromStrInternal(), which was not modified. That gives more weight to the idea that there is some kind of memory issue going on here and one hopes that servers would be less affected. Either way, they largest use cases we have seen are for about 6 million files so if we can improve that case I believe we will be better off.

Further analysis showed that most of the time was taken up writing the size and timestamp fields, which makes almost no sense. The same amount of time was used if they were hard-coded to 0, which points to some odd memory issue on the M1 architecture.

This change has been planned for a while, but the particular impetus at this time is that small file support requires additional fields that would increase manifest memory usage by about 20%, even if the feature is not used.

Note that the Pack code has been updated to use the new varint encoder, but the decoder remains separate because it needs to fetch one byte at a time.
2022-01-21 17:05:07 -05:00
David Steele
575ae77c0d Convert varNewUInt64() to VARUINT64() where possible in manifest.
VARUINT64() does not require memory to be allocated from the mem context so should be used when possible.
2022-01-21 15:57:22 -05:00
David Steele
4a73a02863
Simplify manifest defaults.
Manifest defaults for user, group, and mode were previously generated by scanning the data to find the most common values. This was very accurate but slow and complicated. It could also lead to surprising changes in the manifest when a default value suddenly changed.

Instead, use the $PGDATA path to generate defaults. In the vast majority of cases the same user/group should own all the path/files and the default file mode is easily derived from the path mode. There may be some edge cases where this generates larger manifests, but in general it reduces time and complexity when saving the manifest.

Remove the MCV code since it is longer longer used.
2022-01-21 15:22:48 -05:00
David Steele
b0db4b8ff0 Simplify base path mode in mock/all integration tests.
Change the mode back to 0700 earlier to reduce churn in the expect logs.

This will be especially important in a future commit that gets the defaults exclusively from the base path.
2022-01-21 08:52:51 -05:00
David Steele
8c062e1af8
Remove primary flag from manifest.
This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role.

Since backup is the only code concerned with the primary flag, move the code into the backup module.

We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
David Steele
16559d9e42 Use the PG_FILE_POSTMTRPID constant where appropriate.
Do the same in Perl with the MANIFEST_FILE_POSTMTRPID constant.
2022-01-20 08:41:05 -05:00
David Steele
4ca0590a51 Refactor backupProcessQueue() to use BackupJobData struct.
BackupJobData has several members that backupProcessQueue() needs so it is more efficient to use them rather than passing them separately or getting them from the configuration.
2022-01-20 08:06:42 -05:00
David Steele
e21ba7c92b Remove extra spaces. 2022-01-18 17:40:53 -05:00
David Steele
f84909f85f Fix issue with Posix read offset handling after an error.
Coverity pointed out that -1 could be passed to lseek() (added in a79034ae) after a file failed to open because it is missing. Overall it seems simpler to enclose the success code in an else block to prevent any repeats of this mistake in the future.

This was not an active bug because there are currently no cases where we do read offsets in a file that is allowed to be missing.

Also remove the result flag since it is easier to just check that the file descriptor is valid.
2022-01-18 17:26:30 -05:00
David Steele
91721e391a Fix typo. 2022-01-17 15:39:31 -05:00
David Steele
b791f1c82f Implement restore ownership without updating manifest internals.
Updating the manifest this way was not a great idea because it broke abstraction for the object. This meant certain changes to the interface and internals were not possible because the code was modifying internal manifest data.

Instead track the user replacements entirely in the restore module.

This also has the benefit of eliminating a pass over the manifest path/file/link lists.
2022-01-15 14:33:38 -05:00
Christoph Berg
3097acd73a
Add support for AWS S3 server-side encryption using KMS.
AWS S3 integrates with AWS Key Management Service (AWS KMS) to provide server side encryption of S3 objects. This integration protects objects under encryption keys that never leave AWS KMS unencrypted.
2022-01-13 08:46:14 -05:00
David Steele
92ea3e05fb cfgOptionGroupName() replacement missed in aced5d47. 2022-01-11 19:15:09 -05:00
David Steele
a79034ae2f
Add read range to all storage drivers.
The range feature allows reading out an arbitrary chunk of a file and will be important for efficient small file support.

Now that all drivers are required to support ranges remove the storageFeatureLimitRead feature flag that was implemented only by the Posix driver.
2022-01-11 14:42:53 -05:00
David Steele
2fd100bf12 Rename ConfigOptionGroupData.indexDisplay to indexName.
This is more consistent with the function name and purpose.
2022-01-10 17:40:18 -05:00
David Steele
2cddbbdee0 Remove obsolete cfgOptionHostPort()/cfgOptionIdxHostPort().
These functions were made obsolete by the refactor in 6a124584.
2022-01-10 17:20:48 -05:00
David Steele
7c627c12b7 Refactor option and option group config structs.
This allows individual structs to be assigned to variables to make the code easier to read and perhaps a bit more efficient.
2022-01-10 17:00:58 -05:00
David Steele
aeecb500f5 Improve implementation of cfgOptionIdxName().
Cache option names after they are generated rather than regenerating them each time.
2022-01-10 14:47:29 -05:00
David Steele
aced5d47ed Replace cfgOptionGroupIdxToKey() with cfgOptionGroupName().
Do the replacement anywhere cfgOptionGroupIdxToKey() is being used to construct a group name in a message. cfgOptionGroupName() is better for this case since it also includes the name of the group so that it does not need to be repeated in each message.
2022-01-10 09:10:06 -05:00
David Steele
9657f1b325 Dynamically allocate index to key index map.
Now the config/config module has no notion of max option indexes. The config/parse still needs the max but this will be removed at a later date.
2022-01-09 19:53:05 -05:00
David Steele
e4b48eb430 Fix inconsistent group display names in messages.
In other instances there are no dashes, e.g. repo1 or pg1. Make these messages match.
2022-01-09 19:43:44 -05:00
David Steele
5f78a5fc18 Add ioCopy().
Functionality to copy from IoRead to IoWrite is frequently used so centralize it. This also simplifies coverage testing in places where a loop was required before.
2022-01-09 13:19:43 -05:00
David Steele
47954774c6 Combine encrypted backupFile() tests with unencrypted tests.
This makes it easier to comment out all the tests while developing without getting unused variable errors.
2022-01-09 10:11:00 -05:00
Stefan Fercot
d866dd5c29
Add backup LSNs to info command output.
The backup LSNs are useful for performing LSN-based PITR. LSNs will not be displayed in the general text output (without --set) because they are probably not useful enough to deserve their own line.
2022-01-07 14:09:58 -05:00
David Steele
bb4b30ddd3
Remove support for PostgreSQL 8.3/8.4.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.

It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
2022-01-06 15:34:04 -05:00
Reid Thompson
ef62ef2379 Fix comment missed in 6bd280f7. 2022-01-06 14:24:32 -05:00
Reid Thompson
fdbeb8e7d6 Fix typo in error message. 2022-01-06 14:22:56 -05:00
Reid Thompson
a82f0179cd
Note that replications slots are not restored.
Update documentation and help to note that replication slots are not restored and reference the PostgreSQL documentation to explain why.
2022-01-04 16:11:27 -05:00
David Steele
226cfbdcde Fix typo. 2022-01-04 15:48:00 -05:00
David Steele
591a72c59e Update config.guess and config.sub to latest versions. 2022-01-03 08:57:05 -05:00
David Steele
75e4580919 Begin v2.38 development. 2022-01-03 08:52:25 -05:00
David Steele
f18f2d9991 v2.37: TLS Server
Bug Fixes:

* Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)
* Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)

Features:

* Add TLS server. (Reviewed by Stephen Frost, Reid Thompson, Andrew L'Ecuyer.)
* Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)

Improvements:

* Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)
* Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)
* Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)
* Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)
* Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)
* Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)

Documentation Improvements:

* Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)
* Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)
* Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)
2022-01-03 08:43:55 -05:00
David Steele
62fbee72ad Update LICENSE.txt and PostgreSQL copyright for 2022. 2022-01-01 10:50:16 -05:00
David Steele
d6ebf6e2d6 Remove dead test code. 2021-12-30 18:54:36 -05:00
David Steele
fccb7f7dd4 Add release note regarding IANA approval of the default TLS port. 2021-12-28 17:39:22 -05:00
Reid Thompson
6a12458440
Parse protocol/port in S3/Azure endpoints.
Utilize httpUrlNewParseP() to parse endpoint and port from the URL in the S3 and Azure helpers to avoid issues where protocol was not expected to be part of the URL.
2021-12-16 10:30:59 -05:00
David Steele
f06101de77 Add TLS server documentation.
Add documentation and make the feature visible.
2021-12-16 09:47:04 -05:00
David Steele
615bdff403
Fix socket leak on connection retries.
This leak was caused by the file descriptor variable getting clobbered after a long jump. Mark it as volatile to fix.

Testing this is a bit complex because the issue only happens in optimized builds, if at all. Put the test into the performance suite, which is always optimized, until a better idea presents itself.
2021-12-14 14:53:41 -05:00
David Steele
a73fe4eb96
Fix restore delta link mapping when path/file already exists.
If a path/file was remapped to a link using either --link-map or --link-all there would be no affect if the path/file already existed. If a link existed it would be properly updated and converting a link to a path/file also worked.

The issue happened during delta cleanup, which failed to check if the existing path/file had been remapped to a link.

Add checks for newly mapped path/file links and remove the old path/file we required.
2021-12-10 15:53:40 -05:00
David Steele
19a7ec69de Close expect log file when unit test completes.
This did not cause any issues, but it is better to explicitly close open files.
2021-12-10 15:04:55 -05:00
Christoph Berg
c38e2d3170 Add verb to HTTP error output.
This makes it easier to debug HTTP errors.
2021-12-08 15:00:19 -05:00
David Steele
be4ac3923c
Error when restore is unable to find a backup to match the time target.
This was previously a warning but the warning is easy to miss so a lot of time may be lost restoring and recovering a backup that will not hit the target.

Since this is technically a breaking change, add an "important note" about the change to the release.
2021-12-08 13:57:26 -05:00
Stefan Fercot
6723305937
Add warning when checkpoint_timeout exceeds db-timeout.
In the backup command, add a warning if start-fast is disabled and the PostgreSQL checkpoint_timeout is greater than db-timeout.

In such cases, we might timeout before the checkpoint occurs and the backup really starts.
2021-12-08 12:29:20 -05:00