1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4206 Commits

Author SHA1 Message Date
David Steele
7e5adc0359 Use raw compression/encryption to bundling and block incremental backup.
Raw encryption was already being used for block incremental. This commit adds raw compression to block incremental where possible (see da918587).

Raw compression/encryption is also added to bundling for a backup set when block incremental is enabled on the full backup. This prevents a break in backward compatibility since block incremental is not backward compatible.
2023-03-07 18:46:24 +07:00
David Steele
da91858702 Add optional raw format for compression types.
Raw format saves 12 bytes of header for gzip and 4 bytes of checksum for lz4 (plus CPU overhead). This may not seem like much, but over millions of small files or incremental blocks can really add up. Even though it may be a relatively small percentage of the overall backup size it is still objectively a large amount of data.

Use raw format for protocol compression to exercise the feature.

Raw compression format will be added to bundling and block incremental in a followup commit.
2023-03-07 18:31:17 +07:00
David Steele
f6e307365f Improve interface handling in storage module.
Make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object.

The prior method was more efficient when mem contexts had a much higher cost. Now mem contexts are cheap so it makes more sense to structure the objects in a way that works better with mem context auditing. This also means the mem context does not need to be stored separately since it can be extracted directly from the interface object.

There are other areas that need to get the same improvement before the specialized objMoveContext() and objFreeContext() functions can be removed.
2023-03-07 11:27:41 +07:00
David Steele
120a49b659 Improved error handling in build-code binary.
Show a full stack trace instead of just the error.
2023-03-06 21:07:08 +07:00
David Steele
0818601c05 Set online flag in manifest in command/restore unit tests.
This flag does not currently affect restore behavior but it will in an upcoming commit. Set the flag here to simplify the test diff in the upcoming commit.
2023-03-06 09:49:33 +07:00
David Steele
c656669ac2 Allow control version and WAL magic to be overridden in test harness.
This makes it easier to write tests for invalid control version/WAL magic.

Also add HRN_PG_WAL_TO_BUFFER() to simplify generation of WAL headers.
2023-03-04 12:50:02 +07:00
David Steele
1648c133d6
Keep only one all-default group index.
It is possible for a group index to be created for an option that is later found to not meet dependencies. In this case all values would be default leading to a phantom group, which can be quite confusing.

Remove group indexes that are all default (except the final one) and make sure the key for the final all default group index is 1.
2023-03-04 12:45:08 +07:00
Stefan Fercot
4394479776 Fix typo and remove extraneous linefeed. 2023-02-28 08:47:51 +07:00
David Steele
16ac5ee8d3 Rename block incremental manifest keys.
Since the keys need to be read/written in order, these keys make the logic a bit simpler.
2023-02-26 16:13:44 +07:00
David Steele
a9867cb0b8 Add repo-block-age-map and repo-block-size-map options.
Make these options configurable. This is primarily for testing purposes so the new options will be kept internal.
2023-02-26 14:49:34 +07:00
Christophe Courtois
15d5dcdd3b
Add explicit instructions for upgrading between 2.x versions.
Add an explicit statement that there is nothing special to do when upgrading between 2.x versions.

Leave the previous paragraph about the default location that changed between 2.00 and 2.02, as it is more a matter of transitioning from 1.x to 2.x.
2023-02-26 14:41:32 +07:00
David Steele
dffc933384 Rename DeltaMap to BlockHash.
This more accurately describes what the object does.
2023-02-13 09:17:30 +07:00
David Steele
779efe0d7a Consistently declare block incremental size as size_t.
The block is often allocated in memory so size_t makes more sense than uint64_t.
2023-02-09 13:01:56 +07:00
David Steele
d520816acf Remove parameter list from deltaMapNew().
Since this filter cannot be used remotely (yet) there is no reason to create a parameter list.
2023-02-09 08:11:05 +07:00
David Steele
3feed389a2 Improve IoChunkedRead end-of-file handling.
Determine end-of-file earlier to improve throughput.

Also clean up some comments and formatting.
2023-02-08 22:34:23 +07:00
David Steele
089fae035b Add block incremental to real/all test output. 2023-02-07 14:09:50 +07:00
David Steele
31cad5e09e Check for stray execute permissions in test.pl --code-format.
Sometimes these can get set while updating permissions from with a VM or container.

Also fix a few permissions that were not correct.
2023-02-01 12:57:04 +07:00
David Steele
8e7e9d36a1 Fix contributors in release notes. 2023-01-31 21:28:28 +07:00
David Steele
c5907a2e71 Remove references to SSH made obsolete when TLS was introduced.
Also remove details about SSH compression that are not helpful.
2023-01-31 08:28:32 +07:00
David Steele
9ae6b6ef0f Update comments missed in copy/paste. 2023-01-31 08:04:09 +07:00
David Steele
ce0ea2cfab Use uncrustify for code formatting.
The code is not completely reflowed yet so there are some cases that uncrustify will not catch. The formatting will be improved over time.

Some block of code require special formatting so have been surrounded with the {uncrustify-off}/{uncrustify-on} markers. These exceptions should be kept to a minimum.

Add --code-format (to reformat code) and --code-format-check (to check formatting) to test.pl.

Add a CI test that will check code formatting. Code must be correctly formatted before it can be merge to integration.

Add documentation to the coding standards for code formatting.
2023-01-30 12:22:41 +07:00
David Steele
d4070c9064 Reformat code with uncrustify.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:

* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.

Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.

Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
David Steele
b2202c36d9 Fix formatting errors.
Errors in our current (manually-maintained) code format discovered by uncrustify.
2023-01-30 11:16:31 +07:00
David Steele
a96837d2ed Refactor loop limit in removeExpiredArchive().
This simplifies the formatting, which was a bit tortured. Also, there is no need to check the limit on each iteration so this is a bit more efficient.
2023-01-30 10:55:14 +07:00
David Steele
f91af30584 Add FN_PRINTF() macro.
Bring the format(printf) attribute in line with the FN_NO_RETURN and FN_INLINE_ALWAYS macros.

This is simpler to read and can be customized for different compilers.
2023-01-30 10:51:48 +07:00
David Steele
f8ea51bf8f Refactor stackTraceToZ() to avoid multiple versions of the function.
stackTraceToZ() was split this way in c8264291 to allow complete coverage. 0becb6da added a shim to improve coveage but missed simplifying the function.
2023-01-30 10:27:17 +07:00
David Steele
0d8e08f69c Update config.guess and config.sub to latest versions. 2023-01-30 09:29:07 +07:00
David Steele
240312110c Begin v2.45 development. 2023-01-30 09:27:04 +07:00
David Steele
053468bfb1 v2.44: Remove PostgreSQL 9.0/9.1/9.2 Support
Improvements:

* Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
* Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
* Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)

Documentation Improvements:

* Add warning about enabling "hierarchical namespace" on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
* Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
* Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
* Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
* Add dark mode to the website. (Suggested by Stephen Frost.)
2023-01-30 09:15:44 +07:00
David Steele
a28f3d49c2 Add --no-back-trace option to test.pl.
Running valgrind and backtrace together has been causing tests to timeout in CI, mostly likely due to limited resources. This has not been a problem in normal development environments.

Since it is still important to run backtraces for debugging, split the u22 test that was doing all this work to run coverage and backtrace together and valgrind-only as a separate test. As a bonus these tests run faster separately and since they run in parallel the total execution time is faster.
2023-01-28 18:55:53 +07:00
David Steele
16c625353d Skip test modules in coverage report.
Coverage of the test code is useful during development but it does not seem worth adding it to the core code coverage report.
2023-01-28 16:22:04 +07:00
David Steele
d5ef4ce747 Fix permissions on source files. 2023-01-28 11:01:29 +07:00
David Steele
3aea997df5
Add warning about enabling "hierarchical namespace" on Azure storage.
If this feature is enabled expire will fail since directories need to be deleted separately.

Ideally we would add support for this feature but for now we'll just document the issue.
2023-01-25 10:59:13 +07:00
David Steele
ed818b3186 Add dark mode to the website.
The colors could use more tweaking but at least the website will no longer blind users running dark mode.
2023-01-25 10:35:03 +07:00
David Steele
1da2666a9e Add manifest test harness.
These macros make adding paths/files/etc to a manifest simpler and easier to read.
2023-01-21 14:03:27 +07:00
David Steele
912eec63bb
Block-level incremental backup.
The primary goal of the block incremental backup is to save space in the repository by only storing changed parts of a file rather than the entire file. This implementation is focused on restore performance more than saving space in the repository, though there may be substantial savings depending on the workload.

The repo-block option enables the feature (when repo-bundle is already enabled). The block size is determined based on the file size and age. Very old or very small files will not use block incremental.
2023-01-20 16:48:57 +07:00
David Steele
008a18555c Fix comment indentation and wrapping. 2023-01-20 15:44:01 +07:00
David Steele
bfc9a5c33e Fix incorrect FUNCTION_TEST_END() in hrnPgWalToBuffer(). 2023-01-14 18:01:48 +07:00
David Steele
ccee5c0fb1 Reduce log level of pgVersionFromStr() and pgVersionToStr(). 2023-01-14 17:12:15 +07:00
David Steele
596c62c54e Simply return in pgVersionToStr(). 2023-01-14 15:25:25 +07:00
David Steele
34e4835ff3
Refactor common/ini module to remove callbacks and duplicated code.
The callbacks in iniLoad() made the downstream code more complicated than it needed to be so use an iterator model instead.

Combine the two functions that were used to load the ini data to remove code duplication. In theory it would be nice to use iniValueNext() in the config/parse module rather than loading a KeyValue store but this would mean a big change to the parser, which does not seem worthwhile at this time.
2023-01-12 21:24:28 +07:00
David Steele
9ca492cecf Audit mem contexts returned from functions into the calling context.
It is possible for functions to accidentally leak child contexts into the calling context, which may use a lot of memory depending on the use case and where it happens.

Use the function return type to determine what should be returned and error when something else is returned. Add FUNCTION_AUDIT_*() macros to handle exceptions.

This checking is only performed during unit tests on the code being covered by the specific unit test.

Note that this does not work yet for memory allocations, i.e. memNew(). These are pretty rare so are not as much of an issue and they can be added in the future.
2023-01-12 17:36:57 +07:00
David Steele
de1dfb66ca Refactor logging functions to never allocate memory.
Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case.

This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit.

Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
David Steele
890b9d0093 Remove mostly useless mem context in storageRemoteInfoProtocolPut().
Maybe this once had a deeper purpose but now at least it is just used to avoid a few trivial allocations. If we really wanted to do that a flag would be better but it does not seem worth the trouble.
2023-01-12 15:04:49 +07:00
David Steele
6ec2241e15 Fix leak in protocolServer(). 2023-01-12 14:24:20 +07:00
David Steele
9d42435ffb Fix leaks in db module. 2023-01-12 14:22:09 +07:00
David Steele
7f7d515441 Fix leak in configOptionRemote(). 2023-01-12 14:19:10 +07:00
David Steele
123d5e77ae Minor optimizations on config/parse module. 2023-01-12 14:17:48 +07:00
David Steele
0becb6da31 Enhance libbacktrace to handle incomplete stack traces.
This fills in backtrace info at the bottom of the call stack when the stack trace is incomplete due to testing. This does not affect release builds, which is why it did not make the first cut, but it turns out to be useful for testing and barely changes the release code (when we do release this).

The recursion test in common/error was simplified because it would now return a very large trace.
2023-01-12 10:22:26 +07:00
David Steele
57fc4aaeb9 Update RHEL user guide to PostgreSQL 11/12.
PostgreSQL 10 has expired from the PGDG yum repository.
2023-01-12 09:06:01 +07:00