1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

57 Commits

Author SHA1 Message Date
David Steele
977ec2e307 Integration test improvements for disk and memory efficiency.
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 15:23:07 -05:00
David Steele
686b6f91da Set archive-check option in manifest correctly when offline.
Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 08:27:21 -05:00
David Steele
3b879c2cb3 Filter logged command options based on the command definition.
Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.

Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.
2019-11-14 16:48:41 -05:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
David Steele
4d84820021 Improve performance of info file load/save.
Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load.

This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files.

Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum.

Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests.

Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
David Steele
8fc1d3883b Fix expire not immediately writing into separate file after backup.
Logging stayed in the backup log until the Perl code started.  Fix this so it logs to the correct file and will still work after the Perl code is removed.
2019-08-17 17:43:56 -04:00
David Steele
1708f1d151 Use minio for integration testing.
ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing.  Replace it with minio which starts quickly and ships as a single executable or a tiny container.

Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly.

This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-02 22:20:35 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
Cynthia Shang
b498188f01 Error on db history mismatch when expiring.
Amend commit 434cd832 to error when the db history in archive.info and backup.info do not match.

The Perl code would attempt to reconcile the history by matching on system id and version but we are not planning to migrate that code to C.  It's possible that there are users with mismatches but if so they should have been getting errors from info for the last six months.  It's easy enough to manually fix these files if there are any mismatches in the field.

Contributed by Cynthia Shang.
2019-06-24 11:59:44 -04:00
David Steele
434cd83285 The expire command is implemented entirely in C.
This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests.

Along the way at least one bug was fixed, see issue #748.

Contributed by Cynthia Shang.
2019-06-18 15:19:20 -04:00
David Steele
86482c7db9 Reduce log level for all expect tests to detail.
The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.

Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-22 18:23:44 -04:00
David Steele
e4cc008b98 Remove duplicated hint.
At some point this hint got added to the underlying code so it is no longer needed here.
2019-05-22 11:31:38 -04:00
David Steele
1b48684713 The archive-push command is implemented entirely in C.
This new implementation should behave exactly like the old Perl code with the exception of updated log messages.

Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
David Steele
6d3e18b181 Reduce expect log level in mock/stanza tests.
The expect tests were originally a rough-and-ready type of unit test so monitoring changes in the expect log helped us detect changes in behavior.

Now the stanza code is heavily unit-tested so the detailed logs mainly cause churn and don't have any measurable benefit.

Reduce the log level to DETAIL to make the logs less verbose and volatile, yet still check user-facing log messages.
2019-02-24 06:55:59 +02:00
David Steele
b0b5989aca Migrate remote archive-get command to C.
All required protocol commands are implemented so this is mostly a matter of enabling the feature and updating expect logs.
2019-02-20 22:57:18 +02:00
David Steele
7c2fcb63e4 Enable encryption for archive-get command in C.
The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.

The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
David Steele
6c23830991 Add encryption capability to Info* objects.
Info objects can be stored encrypted and loaded with a cipher type and passphrase.
2018-11-28 14:41:10 -05:00
David Steele
fea27dbd7e Return IoFilterGroup * from ioFilterGroupAdd().
This allows filters adds to be chained.
2018-11-27 22:02:08 -05:00
David Steele
b5690e21a4 Allow I/O read interface to explicitly request blocking reads.
TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.

Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.

In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.
2018-11-23 12:01:36 -05:00
David Steele
d0e553e4c2 Reduce debug level for infoIni() to test.
Getters should generally not be logging at debug or trace level since it clutters the log.

In passing move the destructor to the end of the file.
2018-11-16 16:05:06 -05:00
David Steele
12b3be1d8e Limit usable Buffer size without changing allocated size.
Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.

This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.
2018-11-08 16:48:33 -05:00
David Steele
070455ce44 Correct current history item in InfoPg to always be in position 0.
The InfoPg object was partially modified in 960ad732 to place the current history item in position 0, but infoPgDataCurrent() didn't get updated correctly.

Remove this->indexCurrent and make the current position always equal 0.  Use the new lstInsert() function when adding new history items via infoPgAdd(), but continue to use lstAdd() when loading from a file for efficiency.

This does not appear to be a live bug because infoPgDataCurrent() and infoPgAdd() are not yet used in any production code.  The archive-get command is the only C code using InfoPG and it always looks at the entire list of items rather than just the current item.

Suggested by Cynthia Shang.
2018-10-19 12:31:56 +02:00
David Steele
d038b9a029 Support configurable WAL segment size.
PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB.

There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup.  Both operations require the WAL segment size to properly build a list.

Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database.

The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead.  This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them.  While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL.

Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
Cynthia Shang
b6b2c915b2 Allow hashSize() to run on remote storage.
Apparently we never needed to run this function remotely.

It will be needed by the backup checksum delta feature, so implement it now.

Contributed by Cynthia Shang.
2018-09-18 11:39:48 -04:00
David Steele
c49eaec776 Full abstraction of storage driver interface.
The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly.

Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC).

Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
David Steele
5aa458ffae Simplify debug logging by allowing log functions to return String objects.
Previously, debug log functions had to handle NULLs and truncate output to the available buffer size.  This was verbose for both coding and testing.

Instead, create a function/macro combination that allows log functions to return a simple String object.  The wrapper function takes care of the memory context, handles NULLs, and truncates the log string based on the available buffer size.
2018-09-11 18:32:56 -04:00
David Steele
9b9396c7b7 Migrate local, unencrypted, non-S3 archive-get command to C.
The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs.  Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C.

This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible.  It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
David Steele
9e574a37dc Make archive-get info messages consistent between C and Perl implementations.
The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all.

Temporarily track the async server status with a flag so that info messages are not output in the async process.  The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 12:30:48 -04:00
Cynthia Shang
e351b8c67c Improve info command to display the stanza cipher type.
Contributed by Cynthia Shang.
Suggested by Douglas J Hunley.
2018-09-10 13:09:45 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
91be372e6a Set log-timestamp=n for integration tests.
This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 11:24:38 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
0381945caa Show command parameters as well as command options in initial info message. 2018-04-17 18:47:14 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
Cynthia Shang
dd31ae832d Port most of Config::Config::configLoad() from Perl to C.
Contributed by Cynthia Shang.
2018-03-12 11:25:20 -04:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
David Steele
d35757c05f Fix missing log-level-stderr option for stanza-* commands. 2018-02-05 15:19:28 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists.
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl.
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
1f39a34abc Move lock release later in exitSafe().
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
2018-01-17 10:31:21 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files.
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-18 20:02:54 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions.
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00