1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

2487 Commits

Author SHA1 Message Date
David Steele
7ab07dc580 Validate checksums are set in the manifest on backup/restore.
This is a modest start but it addresses the specific issue that was caused by the bug fixed in 45ec694a.  This validation will produce an immediate error rather than erroring out partway through the restore.

More validations are planned but this is the most important one and seems safest for this release.
2020-01-26 21:58:59 -07:00
David Steele
45ec694af2 Fix missing files corrupting the manifest.
If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored.

The issue was that removing files from the manifest invalidated the pointers stored in the processing queues.  When a file was removed, all the pointers shifted to the next file in the list, causing a file to be unprocessed.  Since the unprocessed file was still in the manifest it would be saved with no checksum, causing a failure on restore.

When process-max was > 1 then the bug would often not express since the file had already been pulled from the queue and updates to the manifest are done by name rather than by pointer.
2020-01-26 13:19:13 -07:00
David Steele
9b47ff2746 Sort last processing queue on backup from standby.
The last queue was not being sorted when a primary queue was added first.

This did not affect the backup or integrity but could lead to slightly lower performance since large files were not always copied first.
2020-01-26 12:29:53 -07:00
David Steele
0444d37414 Remove obsolete include to ../libc. 2020-01-24 10:43:47 -07:00
Marc Cousin
b1c5885017 Add lib path for libpq in case it is in a non-standard location. 2020-01-24 10:40:42 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
b134175fc7 Use designated initializers to initialize structs.
Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc.  While this appears to work in practice, it is a violation of the C specification.  For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits.

Instead use designated initializers to initialize structs.  These guarantee that struct members will be properly initialized even if they are not specified in the initializer.  Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value.

Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly.  All instances of memset() have been removed except in debug/test code where needed.

Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL.

Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
David Steele
cf2024beaf Add XML2_CONFIG environment variable to configure.
This allows the default 'xml2-config' to be replaced with, e.g. 'pkg-config libxml-2.0', for libxml2 configuration.
2020-01-21 18:47:14 -07:00
David Steele
600a51815f Set client_encoding to UTF8 on PostgreSQL connect.
This is the only non-ASCII character encoding we have tested so make sure that's all we get from PostgreSQL.
2020-01-21 18:42:22 -07:00
David Steele
1706c599bd Set encoding to UTF8 for Debian documentation containers.
This allows testing multi-byte encodings in PostgreSQL.
2020-01-21 18:37:43 -07:00
David Steele
382ddfd79d Begin v2.23 development. 2020-01-21 16:43:44 -07:00
David Steele
bae6e1c9e3 v2.22: Bug Fix
Bug Fixes:

* Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
2020-01-21 16:12:33 -07:00
David Steele
94842ccece Fix comment. 2020-01-21 11:59:25 -07:00
David Steele
03d434c7e1 Remove RHEL package patch now that it has been merged upstream.
Also revert 731ffcfb and update ContainerTest.pm for upstream changes.
2020-01-21 11:57:59 -07:00
David Steele
b89e6b7f69 Fix error in timeline conversion.
The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA.
2020-01-21 10:29:46 -07:00
David Steele
c2da9aa907 Fix invalid bufSize().
bufUsed() should be used here to avoid a possible segfault or xml parse error when bufSize() > bufUsed().
2020-01-19 14:54:38 -07:00
David Steele
c630bda1c1 Remove Debian package patch now that it has been merged upstream. 2020-01-19 10:37:08 -07:00
David Steele
d9efbc3698 Add UTF8 strings to manifest and restore tests.
The most likely place to get UTF8 characters is in database names so make sure UTF8 works in the places where database names are processed.
2020-01-18 10:46:48 -07:00
David Steele
19c39fe1e2 Fix redundancy in comment. 2020-01-17 22:54:19 -07:00
David Steele
ec173f12fb Add MEM_CONTEXT_PRIOR() block and update current call sites.
This macro block encapsulates the common pattern of switching to the prior (formerly called old) mem context to return results from a function.

Also rename MEM_CONTEXT_OLD() to memContextPrior().  This violates our convention of macros being in all caps but memContextPrior() will become a function very soon so this will reduce churn.
2020-01-17 13:29:49 -07:00
David Steele
b5fa9951e3 Use MEM_CONTEXT_BEGIN() block in varFree().
We probably arrived at this unusual construction because of the complexity of getting the mem context.  Whether or not this is a good way to store the mem context, it still makes sense to use the standard pattern for switching mem contexts.
2020-01-17 13:08:47 -07:00
David Steele
c6d6b7dbef Use MEM_CONTEXT_NEW_BEGIN() block instead of memContextNew().
A few places were using just memContextNew(), probably because they did not immediately need to create anything in the new context, but it's better if we use the same pattern everywhere, even if it results in a few extra mem context switches.
2020-01-17 11:58:41 -07:00
David Steele
d3be1e41a4 Add braces and spaces for clarity.
This should make it a little easier to see how the TRY macros work together.
2020-01-16 16:23:40 -07:00
David Steele
254f79803f Use MEM_CONTEXT_BEGIN() block instead of memContextSwitch().
This is the standard pattern but was missed here.
2020-01-16 16:20:59 -07:00
David Steele
8068a610d5 Use MEM_CONTEXT_NEW_BEGIN() block.
This pattern makes more sense.  The prior code was probably copy-pasted from code with slightly different requirements.
2020-01-16 16:19:45 -07:00
David Steele
4274fcbf6f Add missing semicolon.
This worked when the FUNCTION_TEST_RETURN_VOID() macro expanded to nothing because of the final semicolon.  If the FUNCTION_TEST_RETURN_VOID() macro expanded to something then there was one semicolon too few.
2020-01-16 14:42:01 -07:00
David Steele
731ffcfb3d Disable RHEL package builds since upstream is broken. 2020-01-16 14:29:04 -07:00
David Steele
e81629b442 Reclassify Perl and LibC code as test/harness.
These were still being included in the core totals but they are no longer used by core.
2020-01-15 13:53:30 -07:00
David Steele
193b78e150 Remove -s from make instructions.
It's a good option but not required so let the user make the choice.

Also combine configure and make into a single line.
2020-01-15 13:40:17 -07:00
David Steele
9414ad2300 Begin v2.22 development. 2020-01-15 13:29:52 -07:00
David Steele
2c0ba0820d v2.21: C Migration Complete
Bug Fixes:

* Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
* Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.

Features:

* pgBackRest is now pure C.
* Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto.)
* Allow path-style URIs in S3 driver.

Improvements:

* The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
2020-01-15 13:21:52 -07:00
David Steele
8d3710b2fe Fix options being ignored by asynchronous commands.
The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc.

Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section.

Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
David Steele
c43ec9d38c Fix function return logging type.
This was working because the stack was being cleaned up by a try block at the call site.
2020-01-15 08:19:32 -07:00
David Steele
f8a29c266e Clarify comment. 2020-01-13 20:36:42 -07:00
David Steele
a7738ebba3 Update comments in command/remote module. 2020-01-13 13:21:28 -07:00
David Steele
a969a860ab Fix misaligned braces. 2020-01-13 11:13:37 -07:00
David Steele
fe263e87b1 Allow path-style URIs in S3 driver.
Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.

Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.
2020-01-12 11:31:06 -07:00
David Steele
069345d339 Add string constants. 2020-01-12 11:00:10 -07:00
David Steele
3f89ecf8d9 Add time to storage ls JSON output.
Time is supported in all drivers with the update to S3 at 61538f93, so it is now possible to add time to the ls command and have it work on all repo types.
2020-01-10 09:39:33 -07:00
David Steele
0fe7bb2ec4 Improve code that updates/removes pg options passed to a remote.
The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated.  For example, dc1e7ca2 added the pg-user option but this code was not updated.

Instead find the options to update/remove dynamically.  The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
David Steele
4c8653fc8b Update inaccurate comments. 2020-01-09 11:56:03 -07:00
Cynthia Shang
2e11389ed4 Fix comment typo. 2020-01-09 11:52:51 -07:00
David Steele
0c5c78e5e1 Make quoting in cfgExeParam() optional.
Parameter lists that are passed directly to exec*() do not need quoting when spaces are present.  Worse, the quotes will not be stripped and the option value will be garbled.

Unfortunately this still does not fix all issues with quoting since we don't know how it might need to be escaped to work with SSH command configuration.  The answer seems to be to pass the options in the protocol layer but that's beyond the scope of this commit.
2020-01-09 09:23:15 -07:00
David Steele
7de5ce23ad Add internal remote-type option.
This option was overloaded on the general type option but it makes sense to split this out since the meaning is pretty different.

Rename the values to conform to current standards, i.e. pg and repo, now that the Perl code won't care anymore.
2020-01-08 18:59:02 -07:00
David Steele
7a1871c341 Fix test log message to match pg-version parameter name.
It was confusing that this part of the log message did not match the parameter name, which made reproducing test failures from CI a little harder.
2020-01-08 09:54:44 -07:00
David Steele
8e1f78b4c8 Remove obsolete Perl code missed in f0ef73db. 2020-01-07 12:20:51 -07:00
David Steele
a84ae6be04 Fix comment typos. 2020-01-06 16:16:48 -07:00
David Steele
61538f932c Parse dates in storageS3InfoList() and storageS3Info().
Previously dates were not being filled by these functions which was fine since dates were not used.

We plan to use dates for the ls command plus it makes sense for the driver to be complete since it will be used as an example.
2020-01-06 15:53:53 -07:00
David Steele
d2fb4f977c Add httpLastModifiedToTime() to parse HTTP last-modified header. 2020-01-06 15:24:49 -07:00
David Steele
a08298ce1b Add basic time management functions.
These are similar to what mktime() and strptime() do but they ignore the local system timezone which saves having to munge the TZ env variable to do time conversions.
2020-01-06 15:18:52 -07:00