1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2208 Commits

Author SHA1 Message Date
David Steele
15d04ca19c Add recursion and json output to the ls command.
These features finally make the ls command practical.

Currently the JSON contains only name, type, and size.  We may add more fields in the future, but these seem like the minimum needed to be useful.
2019-09-12 16:29:50 -04:00
David Steele
e45baa1830 Add sorting, filters, and recursion to storageInfoList().
These are needed for the ls command and are also useful for testing.
2019-09-12 16:03:05 -04:00
David Steele
92365fb801 Disable missing-field-initializers warnings in unit testing.
This warning gives very unpredictable results between compiler versions and seems unrealistic since most of our structs are zeroed for initialization.

This warning has been disabled in the Makefile for a long time.
2019-09-12 15:55:18 -04:00
David Steele
f809d2f008 Ignore apt-get update errors in Travis CI.
Broken vendor packages have been causing builds to break due to an error on apt-get update.

Ignore errors and proceed directory to apt-get install.  It's possible that we'll try to reference an expired package version and get an error anyway, but that seems better than a guaranteed hard error.
2019-09-12 15:16:42 -04:00
David Steele
506c10f7f2 Sort and find improvements to List and StringList objects.
Push the responsibility for sort and find down to the List object by introducing a general comparator function that can be used for both sorting and finding.

Update insert and add functions to return the item added rather than the list.  This is more useful in the core code, though numerous updates to the tests were required.
2019-09-12 12:04:25 -04:00
David Steele
e4a071ce03 Add missing header in common/user module. 2019-09-12 10:56:45 -04:00
David Steele
b53305cc36 Update incorrect pipelining references to connection reuse.
Connection reuse and pipelining are not the same thing and should not have been conflated.

Update comments and release notes to reflect the correct usage.
2019-09-12 08:17:07 -04:00
David Steele
dca5b63f97 Move documentation job first for Travis CI.
Since this job has been running long recently this should improved overall performance when multiple commits are queued up.
2019-09-10 13:06:44 -04:00
David Steele
f4f21d0df7 Add groupIdFromName() and userIdFromName() to user module.
Update StorageWritePosix to use the new functions.

A side effect is that storageWritePosixOpen() will no longer error when the user/group name does not exist.  It will simply retain the original user/group, i.e. the user that executed the restore.

In general this is a feature since completing a restore is more important than setting permissions exactly from the source host.  However, some notification of this omission to the user would be beneficial.
2019-09-10 13:02:05 -04:00
David Steele
f8d0574759 Increase process timeout and emit occasional warnings.
Travis will timeout after 10 minutes with no output.  Emit a warning every 5 minutes to keep Travis alive and increase the total timeout to 20 minutes.

Documentation builds have been timing out a lot recently so hopefully this will help.
2019-09-10 12:29:36 -04:00
David Steele
e043c6b1bc Update comment missed in d957acb3. 2019-09-09 07:37:57 -04:00
David Steele
1049632873 Add user module for managing system users/groups.
Centralize the management of users and groups.

Also update Posix storage driver where users/groups were already in use.
2019-09-08 20:11:51 -04:00
David Steele
d957acb36b Add function to generate PostgreSQL tablespace identifier.
In PostgreSQL >= 9.0 each tablespace data is stored in a specially named directory so different major versions can share the same tablespace path.
2019-09-08 06:53:23 -04:00
David Steele
051128ed9e Add test macro for comparing String to zero-terminated strings.
This macro avoids the need to wrap the first parameter in strPtr() and should improve readability.
2019-09-08 06:47:52 -04:00
David Steele
0a96764cb8 Remove most references to PostgreSQL control and catalog versions.
The control and catalog versions were stored a variety of places in the optimistic hope that they would be useful.  In fact they never were.

We can't remove them from the backup.info and backup.manifest files due to backwards compatibility concerns, but we can at least avoid loading and storing them in C structures.

Add functions to the PostgreSQL interface which will return the control and catalog versions for any supported version of PostgreSQL to allow backwards compatibility for backup.info and backup.manifest.  These functions will be useful in other ways, e.g. generating the tablespace identifier in PostgreSQL >= 9.0.
2019-09-07 18:04:39 -04:00
David Steele
843a602080 Add user/group id to storageInfo().
This is required for some operations on Posix storage.
2019-09-07 15:41:55 -04:00
David Steele
1c7c1042b4 Remove extraneous semicolons and add spaces. 2019-09-07 12:32:25 -04:00
David Steele
2eb1d1fee3 Rename lstRemove() to lstRemoveIdx().
This works better with the names of upcoming functions and seems sensible even in isolation.
2019-09-06 16:50:57 -04:00
David Steele
dde0c2b0c9 Rename INFO_MANIFEST_FILE to MANIFEST_FILE.
This was missed in 5c314df0 when the module was renamed.
2019-09-06 16:43:53 -04:00
David Steele
4d84820021 Improve performance of info file load/save.
Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load.

This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files.

Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum.

Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests.

Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
David Steele
7334f30c35 Add helper function for adding CipherBlock filters to groups.
Reviewed by Cynthia Shang.
2019-09-06 13:35:28 -04:00
David Steele
5c314df098 Rename infoManifest module to manifest.
The manifest is not an info file so if anything it should be called backupManifest.  But that seems too long for such a commonly used object so manifest seems better.

Note that unlike Perl there is no storage manifest method so this stands as the only manifest in the C code, as befits its importance.
2019-09-05 19:53:00 -04:00
David Steele
8df7d68c8d Fix sudo missed in "Build pgBackRest as an unprivileged user".
286a106a updated the documentation to build pgBackRest as an unprivileged user, but the wget command was missed.  This command is not actually run, just displayed, because the release is not yet available when the documentation is built.

Update the wget command to run as the local user.
2019-09-03 18:28:53 -04:00
David Steele
005684bf1f Begin v2.18 development. 2019-09-03 17:53:50 -04:00
David Steele
ce2bf29998 v2.17: C Migrations and Bug Fixes
Bug Fixes:

* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)

Improvements:

* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)

Documentation Bug Fixes:

* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)

Documentation Improvements:

* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
2019-09-03 16:39:32 -04:00
David Steele
0b5720c642 Fix yum.p.o package being installed when custom package specified.
The {[os-type-is-centos]} expression was missing parens which meant "and" expressions built on it would always evaluate true if the os-type was centos6.

Reported by Joe Ayers, John Harvey.
2019-09-03 14:34:49 -04:00
David Steele
7d8068f27b Don't decode manifest data when it is generated on a remote.
Decoding a manifest from the JSON provided by C to the hash required by Perl is an expensive process.  If manifest() was called on a remote it was being decoded into a hash and then immediately re-encoded into JSON for transmission over the protocol layer.

Instead, provide a function for the remote to get the raw JSON which can be transmitted as is and decoded in the calling process instead.

This makes remote manifest calls as fast as they were before 2.16, but local calls must still pay the decoding penalty and are therefore slower.  This will continue to be true until the Perl storage interface is retired at the end of the C migration.

Note that for reasonable numbers of tables there is no detectable difference.  The case in question involved 250K tables with a 10 minute decode time (which was being doubled) on a fast workstation.
2019-09-03 12:30:45 -04:00
David Steele
1e55b87620 Add more commonly-used buffer constants. 2019-09-03 07:49:04 -04:00
David Steele
7ade3fc1c3 Move constants from the infoManifest module to the infoBackup module.
These constants should be kept separate because the implementation of any info file might change in the future and only the interface should be expected to remain consistent.

In any case, infoBackup requires Variant constants while infoManifest uses String constants so they are not shareable.  Modern compilers should combine the underlying const char * constants.
2019-09-02 21:09:43 -04:00
David Steele
3a28b68b8b Disable S3 and encryption on u18 integration tests for mock/all/1.
This test is commonly used for sanity checking but the combination of S3 and encryption makes it hard to use and encourages temporary changes to make it usable.

Acknowledge this and disable S3 and encryption for this test and move them to mock/all/2.
2019-09-02 19:06:12 -04:00
David Steele
05abab5e72 Add OBJECT_DEFINE_MOVE() macro.
The object *Move() functions are very repetitive so replace them with a macro that works similarly to OBJECT_DEFINE_FREE().
2019-08-30 14:36:02 -04:00
David Steele
d1675b7e91 Add ioReadLineParam() to allow return on eof.
ioReadLine() errors on eof because it has previously been used only for protocol reads.

Returning on eof is handy for reading lines from files where eof is not considered an error.
2019-08-28 10:46:54 -04:00
David Steele
a605117a23 Add required header.
Apparently this header is already included everywhere convert.h is included.
2019-08-28 10:36:21 -04:00
Josh Soref
8074ca6a26 Fix typos in variable names.
Contributed by Josh Soref.
2019-08-26 12:30:22 -04:00
Josh Soref
4a88791a0a Fix typos in the release notes.
Contributed by Josh Soref.
2019-08-26 12:29:43 -04:00
Josh Soref
545ccfa878 Fix typos in the documentation.
Contributed by Josh Soref.
2019-08-26 12:26:00 -04:00
Josh Soref
08bac74d38 Fix typo in stack trace message.
Contributed by Josh Soref.
2019-08-26 12:20:51 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
22aa532be1 Add storage tests for files beginning with dots.
Prevent a regression of the issue fixed in f88012ce by adding some tests.
2019-08-26 11:37:21 -04:00
Josh Soref
8c9df8c290 Fix typo in httpClientReponseHeader() function name.
Contributed by Josh Soref.
2019-08-26 11:35:35 -04:00
Josh Soref
d1cc22f886 Fix typo in CFGDEFDATA_OPTION_OPTIONAL_COMMAND_OVERRIDE macro name.
Contributed by Josh Soref.
2019-08-26 11:25:14 -04:00
David Steele
01c2669b97 Fix exclusions for special files.
Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.

Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.

Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.

Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 07:47:54 -04:00
David Steele
2862f480cd Add special file type to storageInfo().
There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.
2019-08-23 07:24:25 -04:00
David Steele
04e4fde573 Remove unneeded strFree() in storagePosixInfoListEntry().
This strFree() was the primary culprit in the performance issue fixed in 9eaeb33c.

Since the parent mem context is now freed regularly, this strFree() performs better, but still adds time so removing it seems best.
2019-08-22 10:28:08 -04:00
David Steele
f88012cef3 Fix regexp to ignore ./.. directories in the Posix driver.
In versions <= 2.15 the old regexp caused any file or directory beginning with . to be ignored during a backup.  This has caused behavioral differences in 2.16 because the new C code correctly excludes ./.. directories.

This Perl code is only used for testing now, but it should still match the output of the C functions.
2019-08-22 10:18:34 -04:00
David Steele
c002a2ce2f Move info file checksum to the end of the file.
Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.

Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.

There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.

Reviewed by Cynthia Shang.
2019-08-21 19:45:48 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
Cynthia Shang
53f27da3a6 Add checkDbConfig() to compare pgBackRest/PostgreSQL configs.
Checking the PostgreSQL-reported path and version against the pgBackRest configuration helps ensure that pgBackRest is operating against the correct cluster.

In Perl this functionality was in the Db object, but check seems like a better place for it in C.

Contributed by Cynthia Shang.
2019-08-21 15:41:52 -04:00
David Steele
8b93fdf349 Add storagePg*() variants to get storage for a specified host id.
Previously the host id to use was pulled from the host-id option or defaulted to 1.

The stanza, check, and backup commands will all need the ability to address a specified pg host, so add functions to make that possible.
2019-08-21 15:26:52 -04:00
Cynthia Shang
fa640f22ad Allow Info* objects to be created from scratch in C.
Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C.

The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created.  Also, add functions needed to initialize each Info* object to a valid state.

Contributed by Cynthia Shang.
2019-08-21 15:12:00 -04:00