1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

2607 Commits

Author SHA1 Message Date
Cynthia Shang
73315268fd Fix typo. 2020-03-19 12:11:20 -04:00
David Steele
d677b07081 Move coverage code to CoverageTest module.
This code needs some work, which will be easier if it is all in one place.
2020-03-19 12:07:51 -04:00
Cynthia Shang
f538da0571 Updates to contributing documentation. 2020-03-18 17:30:36 -04:00
David Steele
26c89b2c8c Improve testing of files that change size during the backup.
Files can change size during a backup so update and add tests to cover the various scenarios more thoroughly.
2020-03-18 13:40:16 -04:00
David Steele
4ec04e5163 Added redacted manifest to testBackupValidate().
The manifest is excellent for validation but including the entire manifest is too noisy and some values are architecture/algorithm dependent.

Output a redacted version that contains the most important information which can be improved on over time.
2020-03-18 10:10:10 -04:00
David Steele
b8cd1b6790 Add TEST_RESULT_STR_Z_KEYRPL() test macro.
This macro will automatically do key replacement before the comparison. This saves the indentation required for an embedded function call.

Possibly TEST_RESULT_Z_KEYRPL() would also be useful but it will be added when needed.
2020-03-18 10:05:08 -04:00
David Steele
f2548f45ce Allow storage reads to be limited by bytes.
The current use case is reading files from the PostgreSQL cluster during backup.

A file may grow during backup but we only need to copy the number of bytes that were reported during the manifest build.  The rest will be rebuilt from the WAL during recovery so copying more is just a waste of space.

Limiting the copy sizes in backup will be part of a future commit.
2020-03-17 18:16:17 -04:00
David Steele
307e741298 Test that shrunk file is backed up correctly.
It's possible, though rare, for a file to shrink during a backup.

There was no issue with the code but having a test is always a good idea.
2020-03-17 16:01:17 -04:00
David Steele
7f1817c8ad Fix typos. 2020-03-17 11:43:38 -04:00
David Steele
9a47b88da3 Add links to custom coverage report.
When multiple files were missing coverage it could be hard to locate the coverage report for a specific file.

Add links for uncovered files to make this easier.

Also move table titles out of the table so they are valid html.
2020-03-16 20:02:36 -04:00
David Steele
d6cea422a0 Make cfgLoadLogSetting() static.
This function was once required by the test harness.
2020-03-16 18:18:30 -04:00
David Steele
f7dac144a6 Reduce variables extern'd by the common/log module in debug builds.
These days it is better to include the module in define.yaml when we need to poke at the internal implementation.

This doesn't quite work for the log test harness, so for now some variables will need to remain extern'd in debug builds.
2020-03-16 18:16:27 -04:00
David Steele
3fbfcba811 Forbid access to /tmp/pgbackrest in the Vagrantfile.
This matches the error that will be thrown in the vm=none test on Travis CI if a unit test writes to /tmp/pgbackrest.
2020-03-16 17:27:01 -04:00
David Steele
46911c64c1 Make storage and logging dry-run aware.
Enhance dry-run support added in 2fa69af8 by forbidding writes in the storage layer and adding prefixes to log messages.

The former will protect against mistakes in dry-run implementations and the latter will make it clear when a command was executed in dry-run mode.

Update expire unit tests with the new log prefix.
2020-03-16 17:24:21 -04:00
Cynthia Shang
2fa69af8da Add --dry-run option to the expire command.
Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.
2020-03-16 13:56:52 -04:00
Cynthia Shang
4608ca473e Fix indentation. 2020-03-16 12:50:37 -04:00
David Steele
4328bc1ac6 Move raw coverage results to test/result/raw path.
These results were stored in the vagrant path along with a full copy of src.

Instead store the raw coverage data in test/result/raw and change source references to the files that already exist in [test-path]/repo.
2020-03-16 08:41:32 -04:00
David Steele
d702249507 Build binaries in the test path rather than the vagrant path.
It makes more sense to build in the test path since many developers won't have a vagrant path. Anyway, it's better not to modify the vagrant path since it belongs to vagrant.

Instead of installing the binary just mount it into the container from where it was built. This saves a bit of time and space.
2020-03-15 10:09:27 -04:00
David Steele
19d975346b Improve stability of command/check test module.
When pgbackrest was present this test behaved unexpectedly.

While the binary is not currently required for this test is might be in the future so fix the test to prevent a regression.
2020-03-15 09:59:22 -04:00
David Steele
da89d0ad61 Add VPATH to Makefile.
This allows builds in a separate directory without copying the source.
2020-03-15 09:53:42 -04:00
David Steele
959dce569b Update code classification and remove XS definition. 2020-03-14 18:30:24 -04:00
David Steele
213cc6e8be Move docker files to test/result. 2020-03-14 15:40:37 -04:00
David Steele
6827e248cd Move coverage results to test/result. 2020-03-14 15:29:42 -04:00
David Steele
75ff25f17f Move profile results to test/result. 2020-03-14 14:50:36 -04:00
David Steele
0f7fe55f72 Build packages on demand only and change build path.
Building packages is not a normal part of development so don't build packages by default. Instead build them in CI as needed.

Do the builds in test/result instead of .vagrant to be friendlier with hosts that are not running vagrant. Anyway, it's probably not a good idea to be creating files in the .vagrant path.
2020-03-14 14:35:09 -04:00
David Steele
5645c91ed5 Add comments to test/.gitignore. 2020-03-14 14:18:22 -04:00
David Steele
b4c94e42b9 Auto-detect presence of clobbered warning.
Not all compilers (e.g. clang) support this warning so only disable it (i.e. -Wno-clobbered) when supported.
2020-03-14 13:06:21 -04:00
David Steele
4cd060b7fe Generate src/build/aclocal.m4 automatically.
This file is required when macros from the autoconf archive are used in configure.ac
2020-03-14 12:48:08 -04:00
David Steele
9e80c5710e Use a checksum to build configure.ac more efficiently.
Building the configure.ac script can take multiple seconds depending on the state of the autoconf cache. Use a checksum to only rebuild when configure.ac has changed no matter how the timestamps have changed.
2020-03-14 12:39:29 -04:00
David Steele
748f9502eb Remove obsolete ignore. 2020-03-14 10:04:49 -04:00
David Steele
237a3da4d6 Configure and make improvements.
Configure:

* Use standard make variables, e.g. CFLAGS, rather than our own, e.g. CINCLUDE
* Add PG_CONFIG var for configuring custom pg_config location
* Don't error if xml_config or pg_config is missing (but error if libs/headers not found)
* Check for zlib.h header
* Check for lz4frame.h header when liblz4 is present

Make:

* Use gcc-style auto dependencies
* Put src list at the top since it is most frequently modified
* Add clean-all target to also remove auto-generated config files
2020-03-13 09:07:57 -04:00
David Steele
810b2a5265 Remove obsolete ignore. 2020-03-12 13:18:08 -04:00
Marc Cousin
4b24a74afb Fix detection of keepalive options on Linux.
This code stanza was not being included on Linux platforms because of a missing header file.

Also update the order of operations and make the timeout calculations more sensible.
2020-03-12 11:38:50 -04:00
David Steele
e9c1569d8e Fix missing cast.
This was a problem in testing when -Wconversion was enabled and TCP_KEEPIDLE was defined.
2020-03-12 10:48:40 -04:00
Marc Cousin
d6ef6c57c3 Add configure host detection to set standards flags correctly.
Linux and MacOS were tolerant of having all standards flags set but BSD was not.

Instead detect the host type and set standards flags as appropriate.
2020-03-12 10:37:19 -04:00
David Steele
fa1f63198d Add scripts required for host detection by src/configure.
Add to the src/build directory so they will be out of the way.

Also add instructions for updating the scripts to RELEASE.md.
2020-03-12 10:03:57 -04:00
David Steele
c3b6bd2e83 Add separators to make configure.ac easier to read. 2020-03-12 09:50:10 -04:00
David Steele
ccdf4bf3cc Add repo path env var to make release commands easier to run. 2020-03-12 09:43:29 -04:00
David Steele
838ef4eca1 Move configure.ac to src/build.
This file is used to generate src/configure and is not required to make pgbackrest since src/configure is updated before distribution.

Move to src/build so it is out of the way.
2020-03-12 09:34:52 -04:00
David Steele
79a6798f39 Remove extra spaces. 2020-03-12 09:28:59 -04:00
David Steele
2ac9c19d4a Fix misleading comment. 2020-03-12 09:28:16 -04:00
David Steele
181fa1fc8b Detect changes in reference.xml for code auto-generation.
Changes to reference.xml can affect the command-line documentation built into the binary so changes must trigger an auto-generated code build during smart builds.
2020-03-12 09:27:44 -04:00
David Steele
0ba8062f5f Get package source files dynamically during package build.
The prior method was to build a special container to hold these files which meant they would get stale on development systems.  On CI the container was always rebuilt so failures would be seen there even when dev seemed to be working.

Instead get the package source when the package is built to ensure it is as up-to-date as possible.

This change was prompted by failures on the Ubuntu 12.04 container while getting the package source, probably due to an ancient version of git.  Package builds are no longer supported on that platform with the addition of lz4 compression so it didn't seem worth fixing.
2020-03-12 08:48:45 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
Cynthia Shang
cc9d7315db Rename flush->flushing to be consistent with usage in other modules. 2020-03-10 14:05:00 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
d3c83453de Add repo-create, repo-get, repo-put, and repo-rm commands.
These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage.

All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00