1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

2614 Commits

Author SHA1 Message Date
Cynthia Shang
6d8d0eeba7 Add pgBackRest version to Info and Manifest objects.
This was not being exposed previously because it is primarily informational, but now it is needed to reconstruct the backup.info file.
2019-10-08 15:37:08 -04:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
a4152a0ea1 Update release instructions with a reminder to install latex.
Latex is no longer installed by default in the Vagrantfile.
2019-10-08 12:29:42 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
77b0c6c993 Add missing spaces. 2019-10-08 10:35:47 -04:00
David Steele
5394893e33 Remove pgPath parameter from pgControlFromFile().
In practice this function is always used with storagePg*() so pgPath is known.
2019-10-03 11:14:22 -04:00
Brad Nicholson
887ec97e07 Document maximum version for auto-stop option. 2019-10-03 06:58:52 -04:00
David Steele
008ec6b05c Begin v2.19 development. 2019-10-01 17:04:26 -04:00
David Steele
4629237d6a v2.18: PostgreSQL 12 Support
Features:

* PostgreSQL 12 support.
* Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Suggested by Stephen Frost, ejberdecia.)
* Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)

Improvements:

* The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Suggested by James Chanco Jr.)
* Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang.)
* Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang.)
2019-10-01 16:02:07 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
Cynthia Shang
6be7e6fde5 Fix misplaced String declarations. 2019-10-01 10:25:08 -04:00
David Steele
309ae66e2f Remove unneeded static declarations and use sizeof() where appropriate. 2019-10-01 08:47:56 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
33ec5a3aac Sort lists where it might help performance.
Most of these lists should be quite small with the exception of the list in get.c, but it doesn't cost much to sort them and may help in corner cases we have not thought of.
2019-09-29 09:05:55 -04:00
David Steele
a58635ac02 Move C performance tests out of unit tests.
Performance tests were being done in unit tests until there was a better place to put them.

Now there is, so move them there.
2019-09-28 14:24:27 -04:00
David Steele
f1ba428fb0 Add performance test capability in C with scaling.
Scaling allows the starting values to be increased from the command-line without code changes.

Also suppress valgrind and assertions when running performance testing.  Optimization is left at -O0 because we should not be depending on compiler optimizations to make our code performant, and it makes profiling more informative.
2019-09-28 14:02:12 -04:00
David Steele
004ff99a2d Identify Perl performance test by appending -perl.
This is intended to differentiate the upcoming C performance tests from the Perl performance tests that will eventually be migrated.
2019-09-28 13:17:21 -04:00
David Steele
cb62bebadf Use bsearch() on sorted lists rather than an iterative method.
bsearch() is far more efficient than an iterative approach except in the most trivial cases.

For now insert will reset the sort order to none and the list will need to be resorted before bsearch() can be used.  This is necessary because item pointers are not stable after a sort, i.e. they can move around.  Until lists are stable it's not a good idea to surprise the caller by mixing up their pointers on insert.
2019-09-28 10:08:20 -04:00
David Steele
d3d2a7cd86 Add line number and fix spacing in TEST_LOG*() macros. 2019-09-28 09:57:06 -04:00
David Steele
afc483ef86 Clarify which timeline should be used for timeline integration test. 2019-09-27 13:37:59 -04:00
David Steele
0a4122c77d Add reviewer. 2019-09-27 13:34:59 -04:00
Cynthia Shang
6355451870 Fix old option name in documentation. 2019-09-27 13:33:14 -04:00
David Steele
d82102d6ef Add explicit promotes to recovery integration tests.
PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead.  Set target_action explicitly so all versions will behave the same way.

This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 13:04:36 -04:00
David Steele
833d0da0d9 Store recovery file name in integration when testing preserve recovery.
This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 12:29:33 -04:00
David Steele
80eb561caf Add missing PostgreSQL 11 control/WAL versions in Perl tests.
These values don't seem to be used for testing but better to be tidy.
2019-09-27 09:45:11 -04:00
David Steele
d6a6d93a04 Add PostgreSQL 12 to u18 container.
This does not add PostgresQL 12 support; it simply adds PostgreSQL 12 to the u18 container for development and testing.
2019-09-27 09:35:59 -04:00
David Steele
03a7bda511 Refactor recovery file generation.
Separate the generation of recovery values and formatting them into recovery.conf format.  This is generally a good idea, but also makes the code ready to deal with a different recovery file in PostgreSQL 12.

Also move the recovery file logic out of cmdRestore() into restoreRecoveryWrite().
2019-09-27 09:19:12 -04:00
David Steele
cf1e96e827 Refactor handling of ignored files in restore delta cleanup.
Replace the boolean logic with a simple list that will be checked to see if the file should be ignored.
2019-09-27 08:50:33 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
8979336819 Comment out unprivileged user documentation.
We haven't had the time to complete this documentation and it has suffered bit rot.

This prevents us from building the docs on PostgreSQL >= 11 so just comment it all out until it can be updated.
2019-09-26 16:02:39 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
Cynthia Shang
92e48c856a Add missing asserts. 2019-09-26 07:27:07 -04:00
David Steele
f2781bdad4 Allow documentation to be built without encryption.
This is mainly useful for testing, where an encrypted repo is a real nuisance.
2019-09-25 16:38:13 -04:00
David Steele
e968acbdd7 Fix outdated comment.
This was probably missed when a new test was added and the timeline was updated.
2019-09-24 16:55:11 -04:00
David Steele
6af64df9b6 Thoughts on repository version 6. 2019-09-23 19:26:24 -04:00
David Steele
71349c89ae Add TEST_TITLE() macro.
This macro displays a title for each test.  A test frequently has multiple parts and it was hard to tell which subparts went together.  We used ad hoc indentation to do this.

Anything that is a not a title is automatically indented so manually indenting is not longer needed.  This should make the tests and the test output easier to read.
2019-09-23 17:56:17 -04:00
David Steele
2fd2fe509f Add TEST_RESULT_LOG*() and TEST_SYSTEM*() macros.
These macros encapsulate the functionality provided by direct calls to harnessLogResult() and system().  They both have _FMT() variants.

The primary advantage is that {[path]}, {[user]}, and {[group]} will be replaced with the test path, user, and group respectively.  This saves a log of strNewFmt() calls and makes the tests less noisy.
2019-09-23 17:20:47 -04:00
David Steele
d3a7055ee5 Only enable test.pl --debug-test-trace option when --debug also enabled.
The other way makes no sense and leads to compile errors since --debug-test-trace requires some code that is only enabled by --debug.
2019-09-23 15:15:04 -04:00
David Steele
072aeb8109 Add LFs to storage list tests.
This makes the diffs easier to read when testing.
2019-09-23 15:10:31 -04:00
David Steele
c969137021 Migrate backup manifest load/save to C.
The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes,
timestamps, etc.  A list of databases is also included for selective restore.

The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that
nothing is missing or corrupt.  It is also useful for reporting, e.g. size of backup, backup time, etc.

For now, migrate enough functionality to implement the restore command.

Reviewed by Cynthia Shang.
2019-09-23 13:50:46 -04:00
David Steele
5b64c93e8b Add local option for cfgExecParam().
cfgExecParam() was originally written to provide options for remote processes.  Remotes processes do not have access to the local config so it was necessary to pass every non-default option.

Local processes on the other hand, e.g. archive-get, archive-get-async, archive-push-async, and local, do have access to the local config and therefore don't need every parameter to be passed on the command-line.  The previous way was not wrong, but it was overly verbose and did not align with the way Perl had worked.

Update cfgExecParam() to accept a local option which excludes options from the command line which can be read from local configs.
2019-09-20 17:50:49 -04:00
David Steele
3f18040aab Rename MANIFEST_FILE to BACKUP_MANIFEST_FILE.
This is a bit more descriptive and avoids a naming conflict in the new C manifest code.
2019-09-20 08:13:36 -04:00
David Steele
174cb7b3af Add strPathAbsolute() and strLstRemoveIdx().
strPathAbsolute() generates an absolute path from an absolute base path and an absolute/relative path.

strLstRemoveIdx() is a support function based on lstRemoveIdx().
2019-09-19 22:42:28 -04:00
David Steele
9421493d09 Allow resets when generating local/remote command options.
Somehow this was left out of the original implementation, but it is particularly important for certain types of restores.
2019-09-19 22:34:12 -04:00
David Steele
e24b2e3f4d Only test timestamps for files.
In general we don't care about path and link times since they are easily recreated when restoring.

So, outside of storageInfo() we don't need to bother testing them.
2019-09-18 22:23:04 -04:00
David Steele
54638e8704 Add strLstRemove().
Allow a string to be removed from a StringList using the underlying lstRemove().
2019-09-18 22:17:22 -04:00
David Steele
a6a42b18a9 Use switch instead of if else tree in jsonFromKvInternal().
This is more efficient but more importantly it is easier to read.
2019-09-18 22:09:49 -04:00
David Steele
60d93df503 Use a callback to feed jobs to ProtocolParallel.
Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed.  We haven't seen this yet, but with backup and restore on the horizon it will become the norm.

Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
David Steele
ce1c7b0252 Ignore write errors when the ls command is writing to stdout.
It's possible (even likely) that the ls output is being piped to something like head which will exit when it gets what it needs and leave us writing to a broken pipe.

It would be better to just ignore the broken pipe error but currently we don't store system error codes.
2019-09-17 21:04:38 -04:00
Cynthia Shang
8675699d08 Add FAQ for time-based Point-in-Time Recovery. 2019-09-16 14:09:53 -04:00