1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2430 Commits

Author SHA1 Message Date
David Steele
74c3842595 Remove errant tabs and fix spacing. 2019-12-19 16:25:46 -05:00
Mike Palmiotto
7dfea6b717 Remove errant tab. 2019-12-19 16:20:11 -05:00
Mike Palmiotto
dc1e7ca22d Add pg-user option.
Specifies the database user name when connecting to PostgreSQL.

If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior.
2019-12-19 11:26:38 -05:00
David Steele
9452084dd1 Fix misspellings of libpq. 2019-12-17 23:32:39 -05:00
David Steele
63a855e2f7 Fix misaligned continuation character. 2019-12-17 23:26:52 -05:00
David Steele
2f7e139534 Error if option prefix and index total are not both defined.
They are both required for indexed options.
2019-12-17 22:35:30 -05:00
David Steele
d780d084b7 Add comments about increasing Vagrantfile disk size. 2019-12-17 21:56:02 -05:00
Mike Palmiotto
d89d9f1c52 Skip vagrant disksize option if no plugin.
Previously, `vagrant up` would bail if no `vagrant-disksize` plugin was
installed. This option is just a nice-to-have, so skip it rather than
bailing.
2019-12-17 21:47:19 -05:00
David Steele
ac35dcac39 Add note about symlink required for contributing documentation to build. 2019-12-17 21:28:40 -05:00
Stephen Frost
3394a50170 Remove unused Perl library missed in f0ef73db. 2019-12-17 21:25:54 -05:00
David Steele
620386f034 Remove integration tests that are now covered in the unit tests.
Most of these tests are just checking that errors are thrown when required.  These are well covered in various unit tests.

The "cannot resume" tests are also well covered in the backup unit tests.

Finally, config warnings are well covered in the config unit tests.

There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-17 20:14:45 -05:00
David Steele
977ec2e307 Integration test improvements for disk and memory efficiency.
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 15:23:07 -05:00
David Steele
ccea30b8d8 Increase memory in ramdisk for Travis CI testing.
The co6 tests were occasionally running out of space so bump up the size of the ramdisk a bit to hopefully prevent this.

A longer term solution would be to disable the trace-level file logs when running on Travis CI since they seem to be using most of the space.
2019-12-14 10:20:23 -05:00
David Steele
6bd280f7bd Don't warn when stop-auto is enabled on PostgreSQL >= 9.6.
PostgreSQL >= 9.6 uses non-exclusive backup which has implicit stop-auto since the backup will stop when the connection is terminated.

The warning was made more verbose in 1f2ce45e but this now seems like a bad idea since there are likely users with mixed version environments where stop-auto is enabled globally.  There's no reason to fill their logs with warnings over a harmless option.  If anything we should warn when stop-auto is explicitly set to false but this doesn't seem very important either.

Revert to the prior behavior, which is to warn and reset when stop-auto is enabled on PostgreSQL < 9.3.
2019-12-14 09:53:50 -05:00
David Steele
03849840b8 Fix handling of \ in filenames.
\ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading.

Use jsonFromStr() to properly quote and escape \.

Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.
2019-12-13 21:33:13 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
1f2ce45e6b The backup command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C except as noted below.

A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.

The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.

For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.

Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.

Resume will now work even if hardlink settings have been changed.

Reviewed by Cynthia Shang.
2019-12-13 17:14:26 -05:00
David Steele
e206093beb Allow end anchor to be excluded in backupRegExp().
This is useful for matching files in the backup history directory which have characters after the backup label.
2019-12-12 18:52:16 -05:00
David Steele
8acfb6adf4 Add pgLsnRangeToWalSegmentList() to convert lsn range to wal segments. 2019-12-12 16:43:34 -05:00
David Steele
81295fd388 Move not found error into walSegmentFind().
This error is also needed in backup so move it here to centralize it.
2019-12-12 16:28:26 -05:00
David Steele
1378d9c58b Fix bad arithmetic in pgLsnToWalSegment().
/ takes precedence over & but the appropriate parens were not provided.

By some bad luck the tests worked either way, so add a new test that only works the correct way to prevent a regression.
2019-12-12 16:21:51 -05:00
David Steele
676be2c773 Add pgWalPath() to return version-specific WAL path.
Also update the manifest module to use the new function.
2019-12-12 16:11:09 -05:00
David Steele
94cb9540f1 Begin v2.21 development. 2019-12-12 09:05:10 -05:00
David Steele
39fc2b7ad6 v2.20: Bug Fixes
Bug Fixes:

* Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)
* Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.
* Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)
2019-12-12 08:20:21 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
db5175915a Use localtime() to format time_t in cvtTimeToZ().
Using gmtime() produced output skewed by the local timezone.

Since this function is currently only used for debug logging this is not a live bug in the field.
2019-12-11 21:51:32 -05:00
Stephen Frost
811ecd758d Make type in comment more specific. 2019-12-11 14:57:32 -05:00
David Steele
0194a98671 Fix archive-push/archive-get when PGDATA is symlinked.
Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked.

If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
David Steele
8c840c28a6 Fix segfault on unexpected EOF in gzip decompression.
If the compressed stream terminated early then the decompression process would get a flush request (NULL input buffer) since the filter was not marked as done.  This could happen on a zero-length or truncated (i.e. invalid) compressed file.

Change the existing assertion to an error to catch this condition in production gracefully.
2019-12-11 08:48:46 -05:00
David Steele
c933f12f9c Remove obsolete --perl-option option.
This option was used when Perl was executed instead of being embedded.

It has been obsolete for a long time so remove it.
2019-12-10 13:28:15 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
d7d663c2b9 Make buildPutDiffers() work with empty files.
If the file was empty the timestamp was updated.  If the file is empty and there is no content then file should not be saved.
2019-12-10 13:02:36 -05:00
David Steele
800d2972b0 Remove stray uint type.
This was probably copied from an example but some compilers don't like it.
2019-12-09 18:28:20 -05:00
David Steele
471d54a738 Add stringz module to define some commonly used strings.
This module will eventually contain various useful zero-terminated string functions.

For now, using NULL_Z instead of strPtr(NULL_STR) avoids a strict aliasing warning on RHEL 6.  This is likely a compiler issue, but adding these constants seems like a good idea anyway and we are not going to get a fix in a gcc that old.
2019-12-09 17:55:20 -05:00
Cynthia Shang
ca33545630 Remove redundant test and move another test. 2019-12-09 14:06:32 -05:00
David Steele
61ad3a44c9 Note that next repo format should use UTC for backup labels. 2019-12-09 09:12:34 -05:00
David Steele
d3132dae26 Add functions for building new manifests.
New manifests are built before a backup is performed.

Reviewed by Cynthia Shang.
2019-12-08 18:43:47 -05:00
David Steele
2cfde18755 Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment(). 2019-12-08 14:19:47 -05:00
David Steele
f517b141fb Update pq harness to play nicely with variable LSNs. 2019-12-08 14:15:23 -05:00
David Steele
d2587250da Add backup functions to Db object.
These functions implement the database backup functionality for all supported versions.
2019-12-07 18:44:06 -05:00
David Steele
8766326da8 Add protocolRemoteFree() to shutdown a specific remote.
Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
David Steele
5175d52e94 Add constant for pg_tblspc path. 2019-12-07 17:42:42 -05:00
David Steele
78b1e05b7e Remove unused Perl module. 2019-12-07 17:39:25 -05:00
David Steele
35a262951a Pq test harness usability and error reporting improvements.
Pq script errors are now printed in test output in case they are being masked by a later error.

Once a script error occurs, the same error will be thrown forever rather than throwing a new error on the next item in the script.

HRNPQ_MACRO_CLOSE() is not required in scripts unless harnessPqScriptStrictSet(true) is called.  Most higher-level tests should not need to run in strict mode.

The command/check test seems to require strict mode but there's no apparent reason why it should.  This would be a good thing to look into at some point.
2019-12-07 17:33:34 -05:00
David Steele
d6479ddd0e Add log replacements to help test non-deterministic log output.
Some log output (e.g. time) is hard to test because the values can change between tests.

Add expressions to replace substrings in the log with predictable values to simplify testing.

This is similar to the log replacement facility available for Perl expect log testing.
2019-12-07 17:15:20 -05:00
David Steele
e4716ee036 Improve diff output in tests.
Always compare expected vs actual (in that order) and give a hint in the error message to indicate what should be added and what removed.
2019-12-07 17:02:41 -05:00
David Steele
8c47ee296a Improve storage harness test callback.
Add ability to omit the root (i.e. dot) path and get real size of compressed files.
2019-12-07 16:55:50 -05:00
David Steele
1b3770e248 Recopy during backup when resumed file is missing or corrupt.
A recopy would occur if the size or checksum was invalid but on error the backup would terminate.

Instead, recopy the resumed file on any error.  If the error is systemic (e.g. network failure) then it should show up again during the recopy.
2019-12-07 09:48:33 -05:00
David Steele
d3f717c892 Storage hardlink and symlink features require path feature.
Since there is only one driver that supports (or is likely to support) links (Posix), require the path feature to make logic in the code simpler.

The checks are added just in case another driver supports links.
2019-12-07 09:26:51 -05:00
David Steele
e632c60525 Fix backup labels in mock/all resume integration tests.
These were not getting updated to match the directory name when the manifests were copied.

The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 11:48:41 -05:00