1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-07 00:35:37 +02:00
Commit Graph

35 Commits

Author SHA1 Message Date
5f78a5fc18 Add ioCopy().
Functionality to copy from IoRead to IoWrite is frequently used so centralize it. This also simplifies coverage testing in places where a loop was required before.
2022-01-09 13:19:43 -05:00
3f7409019d Ensure ASSERT() macro is always available in test modules.
Tests that run without DEBUG for performance did not have ASSERT() and were using CHECK() instead.

Instead ensure that the ASSERT() macro is always available in tests.
2021-11-24 16:09:45 -05:00
bc352fa6a8 Simplify strIdFrom*() functions.
The strIdFrom*() forced the caller to pick an encoding, which led to a number of TRY...CATCH blocks in the code. In practice the caller does not care which encoding is used as long as the string is valid for some encoding.

Update the strIdFrom*() function to try all possible encodings and only throw an error when the string is not valid for any of them.
2021-11-01 10:08:56 -04:00
90f7f11a9f Add missing static keywords in test modules. 2021-10-18 12:22:48 -04:00
0e76ccb5b7 Convert filter param/result to Pack type.
The Pack type is more compact and flexible than the Variant type. The Pack type also allows binary data to be stored, which is useful for transferring the passphrase in the CipherBlock filter.

The primary purpose is to allow more (and more complex) result data to be returned efficiently from the PageChecksum filter. For now the PageChecksum filter still returns the original Variant. Converting the result data will be the subject of a future commit.

Also convert filter types to StringId.
2021-09-22 10:48:21 -04:00
475b57c89b Allow additional memory to be allocated with a mem context.
The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.

Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.

Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.

Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
6789ec420e Add additional checks to performance/storage test.
The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".

To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
2021-08-10 10:37:37 -04:00
d791bb7298 Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros.
This removes a lot of boiler plate where every instance needs to create these interfaces.

Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes.

In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
1ace1ac938 Improve HRN_FORK*() macros.
Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.

Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
76cfbf833d Rename HARNESS_FORK*() macros to HRN_FORK*().
This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
d6797009f8 Add ioFdReadNewOpen() and ioFdWriteNewOpen().
These functions construct and open in one call, which allows them to be used as function parameters.
2021-07-13 11:13:57 -04:00
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
a4f057bb70 Remove comment formatting from TEST_*() macros.
Comment formatting was not used much but it incurred a heavy cost in each macro to process possible formatting.

Remove formatted comments where they did not contain valuable information and replace with strZ(strNewFmt()) otherwise.
2021-05-22 11:28:56 -04:00
b270253a69 Add defines for many test*() getter functions.
A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.

Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.

Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.

In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
7dd01897fd Convert ProtocolStorageType enum to StringId.
Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
6cc521b6b2 Update storage module to use StringIds.
Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.

Also add new config functions and harness config functions to support StringIds.
2021-04-23 13:19:47 -04:00
8844ced384 Refactor common/io/filter module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/io/filter module.
2021-04-12 16:05:40 -04:00
2016fac0d9 Improve protocol handlers.
Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
28301199eb Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*().
When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed.

Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
cbccae05b8 Skip lz4 in performance/storage test when it is not present. 2021-01-24 15:18:02 -05:00
fda105ebd1 Add casts to performance/storage test for 32-bit architectures. 2021-01-24 15:15:50 -05:00
117f03eba1 Prepare configuration module for multi-repository support.
Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).

Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.

Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.

Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.

Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.

Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.

Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
4d22d6eeca Move file descriptor read/write ready into IoRead/IoWrite.
Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea.

This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select().

Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
cde2c756ea Rename handle to fd.
Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd.

There were also plenty of places fd was used so now all uses are consistent.

The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
bfb489a82d Add file name to make performance/storage test more realistic.
Also add timing information.
2020-07-31 16:18:56 -04:00
216a61d936 Move dummy storage driver to test harness.
The dummy driver is the basis for creating test storage drivers so it makes sense to locate it in the harness where all tests can access it.
2020-07-25 08:44:41 -04:00
620a8d17cf Automatic retry for backup, restore, archive-get, and archive-push.
If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors.

Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
f773d909be Improve storage filter performance tests.
Improve the accuracy of the calculations in several areas with better integer expressions.

Make the input buffer size configurable. Previously it was always 1mb, i.e. block size.

Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
a329afd3be Add MD5 hash filter to performance tests. 2020-05-18 19:02:11 -04:00
22ba1f02ce Convert storagePosixNew() to storagePosixNewP().
An upcoming feature requires new parameters for storagePosixNew() and this causes a lot of churn because almost every test creates a Posix storage object. Some refactoring in the tests might reduce this duplication but storagePosixNew() is collecting a lot of parameters so converting to storagePosixNewP() makes sense in any case.

There are relatively few call sites in the core code but they still benefit from better readability after this change.
2020-04-30 11:01:38 -04:00
5e55d58850 Simplify storage driver info and list functions.
The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent.

Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact.

Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs.

Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
da43db3543 Move common/object.h to common/type/object.h.
This header does not contain a type but is used to define types so this seems like a better location.
2020-03-30 20:52:57 -04:00
a29e25a845 Add storage filter performance test.
This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
3d255dce3c Add performance/storage test.
The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote.

The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00