1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Commit Graph

40 Commits

Author SHA1 Message Date
d49907239e Remove double spaces missed in 1bd5530a. 2023-05-03 09:37:59 +03:00
a05bf6bb15 Rename PG_VERSION_*_STR constants to PG_VERSION_*_Z.
This is more consistent with other zero-terminated string constants and also has the benefit of being shorter.
2023-04-16 17:32:24 +03:00
df419b34b5 Improve retry error messages.
Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code.

Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
740c2258e3 Add pg-version-force option for fork integration.
Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on.

This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.
2023-03-09 08:23:15 +07:00
c656669ac2 Allow control version and WAL magic to be overridden in test harness.
This makes it easier to write tests for invalid control version/WAL magic.

Also add HRN_PG_WAL_TO_BUFFER() to simplify generation of WAL headers.
2023-03-04 12:50:02 +07:00
d4070c9064 Reformat code with uncrustify.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:

* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.

Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.

Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
b2202c36d9 Fix formatting errors.
Errors in our current (manually-maintained) code format discovered by uncrustify.
2023-01-30 11:16:31 +07:00
56b55f81e8 Add repository checksum to make verify and resume more efficient.
Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data.

This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum.

Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
2ab845e263 Store manifest checksums in memory more efficiently.
Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions.

There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
77c721eb63 Remove support for PostgreSQL 9.0/9.1/9.2.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.

Remove all logic associated with 9.0/9.1/9.2 and update the tests.

Document the new support policy.

Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
010efffb0c Add hex encode/decoding to decode module.
This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
65d22e4325 Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
571dceefec Add LENGTH_OF() macro.
Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times.

This macro makes the task easier, uses less space, and automatically adjusts when the type changes.
2022-04-07 19:00:15 -04:00
34d649579e Bundle files in the repository during backup.
Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest.

Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented.

Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups.

Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example.

Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling.

One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
8c062e1af8 Remove primary flag from manifest.
This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role.

Since backup is the only code concerned with the primary flag, move the code into the backup module.

We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
e21ba7c92b Remove extra spaces. 2022-01-18 17:40:53 -05:00
01ac6b6cac Autogenerate test system identifiers.
hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL.

Add constants to represent version system ids in tests. These will eventually be auto-generated.

This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
90f7f11a9f Add missing static keywords in test modules. 2021-10-18 12:22:48 -04:00
5701620408 Rename manifest file primary flag in tests. 2021-10-13 19:02:58 -04:00
b7e17d80ea More efficient memory allocation for Strings and String Variants.
The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed.

Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.
2021-10-07 19:43:28 -04:00
6af827cbb1 Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
ae40ed6ec9 Add jobRetry parameter to HRN_CFG_LOAD().
Allow the default of 0 to be overridden to test retry behavior for commands.
2021-10-01 17:15:36 -04:00
39c1c10120 Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.
It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
065a8c8454 Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
ff95eddc36 Update command/verify test to use standard patterns. 2021-06-23 18:09:46 -04:00
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
73885f8c2e Replace hrnLogResult() with TEST_RESULT_LOG/_FMT().
The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
b270253a69 Add defines for many test*() getter functions.
A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.

Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.

Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.

In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
ef63750e0b Add local process shim.
Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.

The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
ae7f0af202 Move PostgreSQL version interface test functions to a test harness.
Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.

Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
2016fac0d9 Improve protocol handlers.
Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
28301199eb Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*().
When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed.

Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
95063f6812 Make --repo optional for remaining commands except stanza-delete.
Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified.

This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.
2021-03-03 09:21:06 -05:00
2e60b93709 Add backup verification to internal verify command.
This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered.

Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
f32eb9b94e Partial multi-repository implementation.
Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.

Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.

Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
97c260bb56 Add TEST_RESULT_STRLST_*() unit test macros.
This avoids the need for strLstJoin() when testing lists.

Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
ad79932ba5 Add internal verify command.
Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet).

The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00