1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

485 Commits

Author SHA1 Message Date
David Steele
5e1291a29f Rename ls command to repo-ls.
This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands.  Since the repo storage can be located lots of places having a common way to list it makes sense.

Prefix with repo- to make the scope of this command clear.

Update documentation to reflect this change.
2020-03-09 16:41:04 -04:00
David Steele
438b957f9c Add infrastructure for multiple compression type support.
Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.

Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).

Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.

Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
David Steele
02aa03d1a2 Remove obsolete methods in pgBackRest::Storage::Storage module.
All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible.

First remove storageDb() and use storageTest() instead.  Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC.  Now the only storage using the LibC interface is storageRepo().

Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed.  Same for owner().

Remove pathSync() because syncs are not required in the tests.  No test data is reused after a crash.

Path create/exists functions should never be explicitly performed on a repo so remove those.  File exists can be implemented by calling info() instead.

Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete.

Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used.  That also means there are no filters returning results so remove all the result code.

Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 14:10:09 -05:00
David Steele
00647c7109 Remove Perl Db module and LibC dependencies.
This was mostly dead code except the DB_BACKUP_ADVISORY_LOCK constant, moved to the real/all test module, and the function that pulls info from pg_control, moved to ExpireEnvTest.pm.
2020-03-06 07:21:17 -05:00
David Steele
2e0fe25650 Remove dependency on LibC hash filter.
Perl provides Digest::SHA for hashing so there is no need to expose this via LibC anymore.
2020-03-05 18:34:59 -05:00
David Steele
eb4347f20b Use static checksums in mock/all integration tests.
Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests.

We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 13:56:20 -05:00
David Steele
91f321fb86 Rename old page*() functions to conform to new conventions.
The general convention now is to prefix PostgreSQL functions with "pg".
2020-03-04 14:24:40 -05:00
David Steele
ee351682da Rename "gzip" to "gz".
"gz" was used as the extension but "gzip" was generally used for function and type naming.

With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code.  Since the extension is standard and we must use it, also use the extension for all naming.
2020-02-27 12:09:05 -05:00
David Steele
9e0dc83e87 Begin v2.25 development. 2020-02-25 17:18:25 -05:00
David Steele
495dec44f0 v2.24: Auto-Select Backup Set for Time Target
Bug Fixes:

* Prevent defunct processes in asynchronous archive commands. (Reviewed by Stephen Frost. Reported by Adam Brusselback, ejberdecia.)
* Error when archive-get/archive-push/restore are not run on a PostgreSQL host. (Reviewed by Stephen Frost. Reported by Jesper St John.)
* Read HTTP content to eof when size/encoding not specified. (Reviewed by Cynthia Shang. Reported by Christian ROUX.)
* Fix resume when the resumable backup was created by Perl. In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully. (Reported by Kacey Holston.)

Features:

* Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. (Contributed by Cynthia Shang.)

Improvements:

* Skip pg_internal.init temp file during backup. (Reviewed by Cynthia Shang. Suggested by Michael Paquier.)
* Add more validations to the manifest on backup. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Prevent lock-bot from adding comments to locked issues. (Suggested by Christoph Berg.)
2020-02-25 17:05:45 -05:00
David Steele
bb45a80d46 Begin v2.24 development. 2020-01-26 22:47:53 -07:00
David Steele
2358d34485 v2.23: Bug Fix
Bug Fixes:

* Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)

Improvements:

* Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
* Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
2020-01-26 22:38:21 -07:00
David Steele
382ddfd79d Begin v2.23 development. 2020-01-21 16:43:44 -07:00
David Steele
bae6e1c9e3 v2.22: Bug Fix
Bug Fixes:

* Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
2020-01-21 16:12:33 -07:00
David Steele
9414ad2300 Begin v2.22 development. 2020-01-15 13:29:52 -07:00
David Steele
2c0ba0820d v2.21: C Migration Complete
Bug Fixes:

* Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
* Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.

Features:

* pgBackRest is now pure C.
* Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto.)
* Allow path-style URIs in S3 driver.

Improvements:

* The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
2020-01-15 13:21:52 -07:00
David Steele
8d3710b2fe Fix options being ignored by asynchronous commands.
The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc.

Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section.

Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
David Steele
fe263e87b1 Allow path-style URIs in S3 driver.
Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.

Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.
2020-01-12 11:31:06 -07:00
David Steele
7de5ce23ad Add internal remote-type option.
This option was overloaded on the general type option but it makes sense to split this out since the meaning is pretty different.

Rename the values to conform to current standards, i.e. pg and repo, now that the Perl code won't care anymore.
2020-01-08 18:59:02 -07:00
David Steele
8e1f78b4c8 Remove obsolete Perl code missed in f0ef73db. 2020-01-07 12:20:51 -07:00
David Steele
011bcb4869 Remove command option from Perl db application name.
The command option will be removed from the C code so it needs to be removed here as well.

This code is now used only for testing so it's not important that it be so precise.
2019-12-29 21:22:18 -07:00
David Steele
33e328abbf Remove unused LibC code.
The code was made obsolete by the migration to C.
2019-12-28 18:30:32 -07:00
Mike Palmiotto
dc1e7ca22d Add pg-user option.
Specifies the database user name when connecting to PostgreSQL.

If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior.
2019-12-19 11:26:38 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
1f2ce45e6b The backup command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C except as noted below.

A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.

The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.

For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.

Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.

Resume will now work even if hardlink settings have been changed.

Reviewed by Cynthia Shang.
2019-12-13 17:14:26 -05:00
David Steele
94cb9540f1 Begin v2.21 development. 2019-12-12 09:05:10 -05:00
David Steele
39fc2b7ad6 v2.20: Bug Fixes
Bug Fixes:

* Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)
* Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.
* Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)
2019-12-12 08:20:21 -05:00
David Steele
c933f12f9c Remove obsolete --perl-option option.
This option was used when Perl was executed instead of being embedded.

It has been obsolete for a long time so remove it.
2019-12-10 13:28:15 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
78b1e05b7e Remove unused Perl module. 2019-12-07 17:39:25 -05:00
David Steele
8dfe0e48e2 Use more general error code when tablespace linked into PGDATA.
The specific error code was not that useful since we also test the error message which contains details of the link error.
2019-12-02 10:49:25 -05:00
David Steele
0933aeb1ad Remove unused reference to use pgBackRest::Backup::File. 2019-11-29 10:56:13 -05:00
David Steele
686b6f91da Set archive-check option in manifest correctly when offline.
Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 08:27:21 -05:00
David Steele
4683557c54 Remove unused reference to pgBackRest::Db. 2019-11-26 17:22:30 -05:00
David Steele
158e439689 Remove obsolete Perl archive code.
This should have been removed in a1c13a50 but was missed.
2019-11-26 17:16:45 -05:00
David Steele
b145c72b5c Update missing manifest warning in BackupInfo.
This brings the Perl message in line with C to reduce expect log churn.
2019-11-25 08:51:28 -05:00
David Steele
01aefc563d Update Perl page checksum expression.
This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.

However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.

Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 07:37:09 -05:00
David Steele
6f65dde8d3 Begin v2.20 development. 2019-11-12 15:56:28 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
Cynthia Shang
ecae5e34e5 Update expire command to use C backup.info reconstruct.
This was still being done in Perl until the C Manifest object was available.
2019-10-08 17:30:33 -04:00
David Steele
008ec6b05c Begin v2.19 development. 2019-10-01 17:04:26 -04:00
David Steele
4629237d6a v2.18: PostgreSQL 12 Support
Features:

* PostgreSQL 12 support.
* Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Suggested by Stephen Frost, ejberdecia.)
* Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)

Improvements:

* The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Suggested by James Chanco Jr.)
* Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang.)
* Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang.)
2019-10-01 16:02:07 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
David Steele
d6a6d93a04 Add PostgreSQL 12 to u18 container.
This does not add PostgresQL 12 support; it simply adds PostgreSQL 12 to the u18 container for development and testing.
2019-09-27 09:35:59 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
Cynthia Shang
56bf9d0566 Update HINT messages to conform to new standard detailed in CODING.md. 2019-09-14 12:21:08 -04:00
David Steele
15d04ca19c Add recursion and json output to the ls command.
These features finally make the ls command practical.

Currently the JSON contains only name, type, and size.  We may add more fields in the future, but these seem like the minimum needed to be useful.
2019-09-12 16:29:50 -04:00
David Steele
005684bf1f Begin v2.18 development. 2019-09-03 17:53:50 -04:00