1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2245 Commits

Author SHA1 Message Date
David Steele
051128ed9e Add test macro for comparing String to zero-terminated strings.
This macro avoids the need to wrap the first parameter in strPtr() and should improve readability.
2019-09-08 06:47:52 -04:00
David Steele
0a96764cb8 Remove most references to PostgreSQL control and catalog versions.
The control and catalog versions were stored a variety of places in the optimistic hope that they would be useful.  In fact they never were.

We can't remove them from the backup.info and backup.manifest files due to backwards compatibility concerns, but we can at least avoid loading and storing them in C structures.

Add functions to the PostgreSQL interface which will return the control and catalog versions for any supported version of PostgreSQL to allow backwards compatibility for backup.info and backup.manifest.  These functions will be useful in other ways, e.g. generating the tablespace identifier in PostgreSQL >= 9.0.
2019-09-07 18:04:39 -04:00
David Steele
843a602080 Add user/group id to storageInfo().
This is required for some operations on Posix storage.
2019-09-07 15:41:55 -04:00
David Steele
1c7c1042b4 Remove extraneous semicolons and add spaces. 2019-09-07 12:32:25 -04:00
David Steele
2eb1d1fee3 Rename lstRemove() to lstRemoveIdx().
This works better with the names of upcoming functions and seems sensible even in isolation.
2019-09-06 16:50:57 -04:00
David Steele
dde0c2b0c9 Rename INFO_MANIFEST_FILE to MANIFEST_FILE.
This was missed in 5c314df0 when the module was renamed.
2019-09-06 16:43:53 -04:00
David Steele
4d84820021 Improve performance of info file load/save.
Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load.

This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files.

Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum.

Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests.

Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
David Steele
7334f30c35 Add helper function for adding CipherBlock filters to groups.
Reviewed by Cynthia Shang.
2019-09-06 13:35:28 -04:00
David Steele
5c314df098 Rename infoManifest module to manifest.
The manifest is not an info file so if anything it should be called backupManifest.  But that seems too long for such a commonly used object so manifest seems better.

Note that unlike Perl there is no storage manifest method so this stands as the only manifest in the C code, as befits its importance.
2019-09-05 19:53:00 -04:00
David Steele
8df7d68c8d Fix sudo missed in "Build pgBackRest as an unprivileged user".
286a106a updated the documentation to build pgBackRest as an unprivileged user, but the wget command was missed.  This command is not actually run, just displayed, because the release is not yet available when the documentation is built.

Update the wget command to run as the local user.
2019-09-03 18:28:53 -04:00
David Steele
005684bf1f Begin v2.18 development. 2019-09-03 17:53:50 -04:00
David Steele
ce2bf29998 v2.17: C Migrations and Bug Fixes
Bug Fixes:

* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)

Improvements:

* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)

Documentation Bug Fixes:

* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)

Documentation Improvements:

* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
2019-09-03 16:39:32 -04:00
David Steele
0b5720c642 Fix yum.p.o package being installed when custom package specified.
The {[os-type-is-centos]} expression was missing parens which meant "and" expressions built on it would always evaluate true if the os-type was centos6.

Reported by Joe Ayers, John Harvey.
2019-09-03 14:34:49 -04:00
David Steele
7d8068f27b Don't decode manifest data when it is generated on a remote.
Decoding a manifest from the JSON provided by C to the hash required by Perl is an expensive process.  If manifest() was called on a remote it was being decoded into a hash and then immediately re-encoded into JSON for transmission over the protocol layer.

Instead, provide a function for the remote to get the raw JSON which can be transmitted as is and decoded in the calling process instead.

This makes remote manifest calls as fast as they were before 2.16, but local calls must still pay the decoding penalty and are therefore slower.  This will continue to be true until the Perl storage interface is retired at the end of the C migration.

Note that for reasonable numbers of tables there is no detectable difference.  The case in question involved 250K tables with a 10 minute decode time (which was being doubled) on a fast workstation.
2019-09-03 12:30:45 -04:00
David Steele
1e55b87620 Add more commonly-used buffer constants. 2019-09-03 07:49:04 -04:00
David Steele
7ade3fc1c3 Move constants from the infoManifest module to the infoBackup module.
These constants should be kept separate because the implementation of any info file might change in the future and only the interface should be expected to remain consistent.

In any case, infoBackup requires Variant constants while infoManifest uses String constants so they are not shareable.  Modern compilers should combine the underlying const char * constants.
2019-09-02 21:09:43 -04:00
David Steele
3a28b68b8b Disable S3 and encryption on u18 integration tests for mock/all/1.
This test is commonly used for sanity checking but the combination of S3 and encryption makes it hard to use and encourages temporary changes to make it usable.

Acknowledge this and disable S3 and encryption for this test and move them to mock/all/2.
2019-09-02 19:06:12 -04:00
David Steele
05abab5e72 Add OBJECT_DEFINE_MOVE() macro.
The object *Move() functions are very repetitive so replace them with a macro that works similarly to OBJECT_DEFINE_FREE().
2019-08-30 14:36:02 -04:00
David Steele
d1675b7e91 Add ioReadLineParam() to allow return on eof.
ioReadLine() errors on eof because it has previously been used only for protocol reads.

Returning on eof is handy for reading lines from files where eof is not considered an error.
2019-08-28 10:46:54 -04:00
David Steele
a605117a23 Add required header.
Apparently this header is already included everywhere convert.h is included.
2019-08-28 10:36:21 -04:00
Josh Soref
8074ca6a26 Fix typos in variable names.
Contributed by Josh Soref.
2019-08-26 12:30:22 -04:00
Josh Soref
4a88791a0a Fix typos in the release notes.
Contributed by Josh Soref.
2019-08-26 12:29:43 -04:00
Josh Soref
545ccfa878 Fix typos in the documentation.
Contributed by Josh Soref.
2019-08-26 12:26:00 -04:00
Josh Soref
08bac74d38 Fix typo in stack trace message.
Contributed by Josh Soref.
2019-08-26 12:20:51 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
22aa532be1 Add storage tests for files beginning with dots.
Prevent a regression of the issue fixed in f88012ce by adding some tests.
2019-08-26 11:37:21 -04:00
Josh Soref
8c9df8c290 Fix typo in httpClientReponseHeader() function name.
Contributed by Josh Soref.
2019-08-26 11:35:35 -04:00
Josh Soref
d1cc22f886 Fix typo in CFGDEFDATA_OPTION_OPTIONAL_COMMAND_OVERRIDE macro name.
Contributed by Josh Soref.
2019-08-26 11:25:14 -04:00
David Steele
01c2669b97 Fix exclusions for special files.
Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.

Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.

Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.

Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 07:47:54 -04:00
David Steele
2862f480cd Add special file type to storageInfo().
There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.
2019-08-23 07:24:25 -04:00
David Steele
04e4fde573 Remove unneeded strFree() in storagePosixInfoListEntry().
This strFree() was the primary culprit in the performance issue fixed in 9eaeb33c.

Since the parent mem context is now freed regularly, this strFree() performs better, but still adds time so removing it seems best.
2019-08-22 10:28:08 -04:00
David Steele
f88012cef3 Fix regexp to ignore ./.. directories in the Posix driver.
In versions <= 2.15 the old regexp caused any file or directory beginning with . to be ignored during a backup.  This has caused behavioral differences in 2.16 because the new C code correctly excludes ./.. directories.

This Perl code is only used for testing now, but it should still match the output of the C functions.
2019-08-22 10:18:34 -04:00
David Steele
c002a2ce2f Move info file checksum to the end of the file.
Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.

Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.

There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.

Reviewed by Cynthia Shang.
2019-08-21 19:45:48 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
Cynthia Shang
53f27da3a6 Add checkDbConfig() to compare pgBackRest/PostgreSQL configs.
Checking the PostgreSQL-reported path and version against the pgBackRest configuration helps ensure that pgBackRest is operating against the correct cluster.

In Perl this functionality was in the Db object, but check seems like a better place for it in C.

Contributed by Cynthia Shang.
2019-08-21 15:41:52 -04:00
David Steele
8b93fdf349 Add storagePg*() variants to get storage for a specified host id.
Previously the host id to use was pulled from the host-id option or defaulted to 1.

The stanza, check, and backup commands will all need the ability to address a specified pg host, so add functions to make that possible.
2019-08-21 15:26:52 -04:00
Cynthia Shang
fa640f22ad Allow Info* objects to be created from scratch in C.
Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C.

The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created.  Also, add functions needed to initialize each Info* object to a valid state.

Contributed by Cynthia Shang.
2019-08-21 15:12:00 -04:00
David Steele
aa6f7eb862 Fix comment indents. 2019-08-21 14:14:30 -04:00
Cynthia Shang
71d96531f8 Fix comment typo.
Contributed by Cynthia Shang.
2019-08-21 12:10:09 -04:00
Cynthia Shang
c4533765fb Add Db object getters missed in e4901d50.
dbOpen() loaded the version and data directory but there was no way to get the data externally.

Contributed by Cynthia Shang.
2019-08-21 11:59:51 -04:00
Cynthia Shang
27e8235812 Add repoIsLocalVerify() to verify repository locality.
Some commands can only be run on a host where the repository is local.  This function centralizes the check and error.

Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
Cynthia Shang
6a09d9294d Require storage when calling pgControlFromFile().
Previously storageLocal() was being used internally but loading pg_control from remote storage is often required.

Also, storagePg() is more appropriate than storageLocal() for all current usage.

Contributed by Cynthia Shang.
2019-08-21 11:29:30 -04:00
David Steele
286a106ae4 Build pgBackRest as an unprivileged user.
pgBackRest was being built by root in the documentation which is definitely not best practice.

Instead build as the unprivileged default container user.  Sudo privileges are still required to install.

Suggested by Laurenz Albe.
2019-08-20 09:46:29 -04:00
David Steele
6b5366a663 Remove Perl info command invocation missed in 205525b6. 2019-08-19 21:57:14 -04:00
David Steele
f6aef6e466 Properly reset conflicting pg-* options for the remote protocol.
The pg1-socket-path and pg1-port options were not being reset when options from a higher index were being pushed down for processing by a remote.  Since remotes only talk to one cluster they always use the options in index 1.  This requires moving options from the original index to 1 before starting the remote.  All options already set on index 1 must be removed if they are not being overwritten.
2019-08-19 21:45:54 -04:00
David Steele
9eaeb33c88 Fix slow manifest build for very large quantities of tables/segments.
storagePosixInfoList() processed each directory in a single memory context.  If the directory contained hundreds of thousands of files processing became very slow due to the number of allocations.

Instead, reset the memory context every thousand files to minimize the number of allocations active at once, improving both speed and memory consumption.

Reported by Jens Wilke.
2019-08-19 21:36:01 -04:00
David Steele
d411321d28 Add reset to temp memory contexts to save memory and processing time.
Processing large datasets in a memory context can lead to high memory usage and long allocation times.  Add a new MEM_CONTEXT_TEMP_RESET_BEGIN() macro that allows temp allocations to be automatically freed after N iterations.
2019-08-19 21:16:10 -04:00
David Steele
7d97d49f41 Add MostCommonValue object.
Calculate the most common value in a list of variants.  If there is a tie then the first value passed to mcvUpdate() wins.

mcvResult() can be called multiple times because it does not end processing, but there is a cost to calculating the result each time
since it is not stored.
2019-08-18 20:46:34 -04:00
David Steele
8aa1e552b0 Add backup type conversion functions.
Convert back and forth between the string and enum representations of backup types.
2019-08-18 20:09:44 -04:00
David Steele
3df075bf40 Fix test writing "null" into manifest files.
"null" is not allowed in the manifest format (null values should be missing instead) but Perl was treating the invalid values written by this test as if they were missing.

Update the test code to remove the values rather than setting them to "null".
2019-08-18 15:29:18 -04:00