1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2820 Commits

Author SHA1 Message Date
David Steele
ace41d57d1 Clarify that gzip is always used to compress history files. 2020-02-25 09:34:27 -05:00
David Steele
cc743f2e04 Skip pg_internal.init temp file during backup.
If PostgreSQL crashes it can leave behind a pg_internal.init temp file with the pid as the extension, as discussed in https://www.postgresql.org/message-id/flat/20200131045352.GB2631%40paquier.xyz#7700b9481ef5b0dd5f09cc410b4750f6.  On restart this file is not cleaned up so it can persist for the lifetime of the cluster or until another process with the same id happens to write pg_internal.init.

This is arguably a bug in PostgreSQL, but in any case it makes sense not to backup this file.
2020-02-21 11:51:39 -05:00
David Steele
48d0f77fe3 Remove dead LibC macros.
These macros were made obsolete when code was removed from LibC after the C migration was completed.
2020-02-21 11:31:31 -05:00
David Steele
dfc5f67233 Fix typo. 2020-02-17 17:12:22 -06:00
David Steele
ea0af890d8 Reclassify release note to documentation improvement. 2020-02-12 17:27:44 -07:00
David Steele
c6b89d74ec Add reviewer. 2020-02-12 17:20:21 -07:00
David Steele
6353e9428d Error when archive-get/archive-push/restore are not run on a PostgreSQL host.
This error was lost during the migration to C.  The error that occurred instead (generally an SSH auth error) was hard to debug.

Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands.  reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
David Steele
dac8119bf1 Add pgIsLocalVerify().
This functionality is required in commands other than restore, so centralize it.
2020-02-12 15:47:07 -07:00
David Steele
e2c304d473 Prevent defunct processes in asynchronous archive commands.
The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.

In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.

Also update the comment on chdir() to more accurately reflect what is happening.

Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).
2020-02-12 12:17:23 -07:00
David Steele
1fa3ae2fcd Prevent lock-bot from marking locked issues as "resolved".
This is certainly not true in all cases, e.g. an issue may be closed if it is added to the backlog.
2020-02-11 19:54:17 -07:00
David Steele
1be9e6854e Prevent lock-bot from adding comments to locked issues.
This will hopefully prevent users from getting notifications when an issue is locked.
2020-02-11 19:52:23 -07:00
David Steele
43936c58a8 Fix resume when the resumable backup was created by Perl.
In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully.
2020-02-11 19:44:06 -07:00
David Steele
44adf21c83 Consolidate archive async exec code.
Move duplicated code to the common module.  This will reduce copy and paste between the get and push modules when changes are made.
2020-02-10 21:30:43 -07:00
David Steele
0eaedc9a6a Improve async archive error file removal.
2a06df93 removed the error file so an old error would not be reported before the async process had a chance to try again.  However, if the async process was already running this might lead to a timeout error before reporting the correct error.

Instead, remove the error files once we know that the async process will start, i.e. after the archive lock has been acquired.

This effectively reverts 2a06df93.
2020-02-10 19:17:11 -07:00
David Steele
8cfbc294fc Fix incorrect error code. 2020-02-10 18:48:47 -07:00
David Steele
1ce71b1e9b Add missing linefeed. 2020-02-10 17:44:39 -07:00
David Steele
71b4cc56cb Rename confessOnError to throwOnError.
Confess is awfully Perl-ish and was likely copied verbatim during the migration.  Rename to what we do now, i.e. throw.
2020-02-06 21:11:15 -08:00
David Steele
2a06df93f3 Remove async archive error file when not throwing an error.
This ensures that the error will not be thrown before the async process has a chance to retry.
2020-02-06 20:59:04 -08:00
David Steele
3721e57a0e Clarify why some recovery options are not commented out for PG >= 12. 2020-02-06 18:28:54 -08:00
Mike Palmiotto
efff54490f Fix release note typo. 2020-02-04 21:19:21 -08:00
David Steele
296aec03be Update contributor name. 2020-01-31 07:50:03 -07:00
David Steele
0f8ec3e478 Read HTTP content to eof when size/encoding not specified.
Generally, the content-size or content-encoding headers will be used to specify how much content should be expected.

There is a special case where the server sends 'Connection:close' without the content headers and the content may be read up until eof.

This appears to be an atypical usage but it is required by the specification.
2020-01-30 14:51:26 -07:00
Cynthia Shang
856980ae99 Auto-select backup set on restore when time target is specified.
Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.

Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 14:38:05 -07:00
Cynthia Shang
f46d1fa74c Add timezone calculations to time module.
Add tzPartsValid() and tzOffsetSecond() to calculate timezone offsets from user provided values.

Update epochFromParts() to accept a timezone offset in seconds.
2020-01-30 11:28:30 -07:00
Cynthia Shang
dbaa5e3473 Add linefeeds to function declarations. 2020-01-29 08:21:36 -07:00
David Steele
80687cbe74 Free TLS connection in common/io-http test.
The test that checks for no output from the server was leaving a connection open which valgrind was complaining about.

Wait on the server long enough to cause the error on the client then close the connection to free the memory.
2020-01-28 10:19:58 -07:00
David Steele
846efaa40f Revert 'Add lib path for libpq in case it is in a non-standard location.`
Putting this before AC_CHECK_LIB breaks on many systems because the location of pg_config is not yet known.
2020-01-28 07:36:20 -07:00
Cynthia Shang
324f7cebe0 Designated initializer cleanup.
Cleanup designated initializers created in b134175f by moving struct members in or out for clarity.
2020-01-27 17:50:07 -07:00
David Steele
24d2494c82 Fix incomplete comment. 2020-01-27 11:25:24 -07:00
David Steele
0a845214a1 Fix typo. 2020-01-26 23:10:29 -07:00
David Steele
697150eaf8 Add more validations to the manifest on backup.
Validate that checksums exist for zero size files.  This means that the checksums for zero size files are explicitly set by backup even though they'll always be the same.  Also validate that zero length files have the correct checksum.

Validate that repo size is > 0 if size is > 0.  No matter what compression type is used a non-zero amount of data cannot be stored in zero bytes.
2020-01-26 23:07:07 -07:00
David Steele
bb45a80d46 Begin v2.24 development. 2020-01-26 22:47:53 -07:00
David Steele
2358d34485 v2.23: Bug Fix
Bug Fixes:

* Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)

Improvements:

* Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
* Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
2020-01-26 22:38:21 -07:00
David Steele
7ab07dc580 Validate checksums are set in the manifest on backup/restore.
This is a modest start but it addresses the specific issue that was caused by the bug fixed in 45ec694a.  This validation will produce an immediate error rather than erroring out partway through the restore.

More validations are planned but this is the most important one and seems safest for this release.
2020-01-26 21:58:59 -07:00
David Steele
45ec694af2 Fix missing files corrupting the manifest.
If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored.

The issue was that removing files from the manifest invalidated the pointers stored in the processing queues.  When a file was removed, all the pointers shifted to the next file in the list, causing a file to be unprocessed.  Since the unprocessed file was still in the manifest it would be saved with no checksum, causing a failure on restore.

When process-max was > 1 then the bug would often not express since the file had already been pulled from the queue and updates to the manifest are done by name rather than by pointer.
2020-01-26 13:19:13 -07:00
David Steele
9b47ff2746 Sort last processing queue on backup from standby.
The last queue was not being sorted when a primary queue was added first.

This did not affect the backup or integrity but could lead to slightly lower performance since large files were not always copied first.
2020-01-26 12:29:53 -07:00
David Steele
0444d37414 Remove obsolete include to ../libc. 2020-01-24 10:43:47 -07:00
Marc Cousin
b1c5885017 Add lib path for libpq in case it is in a non-standard location. 2020-01-24 10:40:42 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
b134175fc7 Use designated initializers to initialize structs.
Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc.  While this appears to work in practice, it is a violation of the C specification.  For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits.

Instead use designated initializers to initialize structs.  These guarantee that struct members will be properly initialized even if they are not specified in the initializer.  Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value.

Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly.  All instances of memset() have been removed except in debug/test code where needed.

Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL.

Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
David Steele
cf2024beaf Add XML2_CONFIG environment variable to configure.
This allows the default 'xml2-config' to be replaced with, e.g. 'pkg-config libxml-2.0', for libxml2 configuration.
2020-01-21 18:47:14 -07:00
David Steele
600a51815f Set client_encoding to UTF8 on PostgreSQL connect.
This is the only non-ASCII character encoding we have tested so make sure that's all we get from PostgreSQL.
2020-01-21 18:42:22 -07:00
David Steele
1706c599bd Set encoding to UTF8 for Debian documentation containers.
This allows testing multi-byte encodings in PostgreSQL.
2020-01-21 18:37:43 -07:00
David Steele
382ddfd79d Begin v2.23 development. 2020-01-21 16:43:44 -07:00
David Steele
bae6e1c9e3 v2.22: Bug Fix
Bug Fixes:

* Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
2020-01-21 16:12:33 -07:00
David Steele
94842ccece Fix comment. 2020-01-21 11:59:25 -07:00
David Steele
03d434c7e1 Remove RHEL package patch now that it has been merged upstream.
Also revert 731ffcfb and update ContainerTest.pm for upstream changes.
2020-01-21 11:57:59 -07:00
David Steele
b89e6b7f69 Fix error in timeline conversion.
The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA.
2020-01-21 10:29:46 -07:00
David Steele
c2da9aa907 Fix invalid bufSize().
bufUsed() should be used here to avoid a possible segfault or xml parse error when bufSize() > bufUsed().
2020-01-19 14:54:38 -07:00
David Steele
c630bda1c1 Remove Debian package patch now that it has been merged upstream. 2020-01-19 10:37:08 -07:00