1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

217 Commits

Author SHA1 Message Date
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
3796b74dca Use stock PostgreSQL page checksum implementation.
We were using a customized version which worked fine but was hard to merge with upstream changes.  Now this code is maintained much like the types in static.auto.h that we copy and check with each release.

The goal is to eventually build directly against PostgreSQL (either source or libcommon) and this brings us one step closer.
2020-03-05 14:23:01 -05:00
David Steele
977ec2e307 Integration test improvements for disk and memory efficiency.
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 15:23:07 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00
David Steele
a52faf83a5 Disable code generation on dry-run. 2019-10-17 11:56:45 +02:00
David Steele
f3b2189659 Remove package build sudo into the container.
By running this in the container we no longer need sudo on the host system for package builds.
2019-10-15 13:27:03 +02:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
642ce003c8 Don't autogenerate embedded libc code by default.
This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.

Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.

Suggested by Stephen Frost.
2019-10-11 11:32:51 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
696e6a7c44 Don't require sudo to run tests with --vm=none.
Run these tests without sudo privileges on Travis to prevent regressions.
2019-10-10 11:21:09 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
f1ba428fb0 Add performance test capability in C with scaling.
Scaling allows the starting values to be increased from the command-line without code changes.

Also suppress valgrind and assertions when running performance testing.  Optimization is left at -O0 because we should not be depending on compiler optimizations to make our code performant, and it makes profiling more informative.
2019-09-28 14:02:12 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
8edc57b581 Remove RHEL package patch merged to upstream. 2019-08-18 15:19:10 -04:00
David Steele
fe196cb0df Update RHEL patch for upstream changes.
This isn't correct, but we can't test anything else if the package doesn't build.
2019-08-09 14:52:26 -04:00
David Steele
5e1ed2e8a5 Remove clang static analysis.
This analysis never produced anything but false positives (var might be NULL) but took over a minute per test run and added 600MB to the test container.
2019-07-05 18:34:15 -04:00
David Steele
9836578520 Remove perl critic and coverage.
No new Perl code is being developed, so these tools are just taking up time and making migrations to newer platforms harder.  There are only a few Perl tests remaining with full coverage so the coverage tool does not warn of loss of coverage in most cases.

Remove both tools and associated libraries.
2019-07-05 16:55:17 -04:00
David Steele
0a96a2895d Add storage layer for tests and documentation.
The tests and documentation have been using the core storage layer but soon that will depend entirely on the C library, creating a bootstrap problem (i.e. the storage layer will be needed to build the C library).

Create a simplified Posix storage layer to be used by documentation and the parts of the test code that build and execute the actual tests.  The actual tests will still use the core storage driver so they can interact with any type of storage.
2019-06-17 09:16:44 -04:00
David Steele
388ba0458c Fix build.flags being removed on each build.
This was being removed by rsync which forced a full build even when a partial should have been fine.  Rewrite the file after the rsync so it is preserved.
2019-05-31 18:37:31 -04:00
David Steele
5c1d4bcd0d Automate coverage summary report generation.
This report replaces the lcov report that was generated manually for each release.

The lcov report was overly verbose just to say that we have virtually 100% coverage.
2019-05-15 13:04:56 -04:00
David Steele
027c263871 Add configure script for improved multi-platform support.
Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile.

Each C file must include "build.auto.conf" before all other includes and defines.  This is enforced by test.pl for includes, but it won't detect incorrect define ordering.

Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
David Steele
1ae8a6a716 Add build-max option to set max build processes.
Currently this controls make processes via -j.
2019-04-23 20:52:03 -04:00
David Steele
18b62a4220 Only run test-level stack trace by default for unit-tested modules.
This amends 70c30dfb which disabled test tracing in general.

Instead, only enable test tracing by default for modules that are being unit tested. This saves lots of time but still ensures that test tracing is working and helps with debugging in unit tests.

Also rename the option to --debug-test-trace for a clarity.
2019-02-27 17:09:19 +02:00
David Steele
70c30dfb61 Disable test-level stack trace by default.
Detailed stack traces for low-level functions (e.g. strCat, bufMove) can be very useful for debugging but leaving them on for all tests has become quite burdensome in terms of time.  Complex operations like generating JSON on a large KevValue can lead to timeouts even with generous values.

Add a new param, --debug-trace, to enable test-level stack trace, but leave it off by default.
2019-02-22 11:40:30 +02:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
22ecbc153a New, concise coverage report for C.
The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.

The new report presents all missing coverage on a single page and excludes code that is covered for brevity.
2018-11-11 17:32:42 -05:00
David Steele
7de9584435 Ignore deleted files in rsync to test/repo.
Deleted files are showing up in git ls-files (added 57d78092) but they don't actually exist on disk.

If there is someway to exclude deleted files from ls-files then I can't find it, so just tell rsync to ignore missing files.
2018-11-04 08:39:48 -05:00
David Steele
57d7809297 Improve efficiency of code generation.
Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.

Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.

The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.
2018-11-03 19:52:46 -04:00
David Steele
7794ab50dc Preserve contents of C unit test build directory between test.pl executions.
The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.

Save the important build flags in a file so the new execution knows whether the build contents can be reused.
2018-11-02 11:56:13 -04:00
David Steele
1a98bd41b4 Mount tmpfs in Vagrantfile instead test.pl.
Mounting/unmounting tmpfs on /home/[user]/test takes time, forces at least 3GB of memory to be available for tests, and makes it harder to preserve data between tests.

Instead, move mounting of tmpfs to the Vagrantfile and add it to fstab so it survives reboots.
2018-11-02 08:37:27 -04:00
David Steele
f0ed89f21f Allow C or Perl coverage to run on more than one VM.
C or Perl coverage tests can now be run on any VM provided a recent enough version of Devel::Cover or lcov is available.

For now, leave u18 as the only VM to run coverage tests due to some issues with older versions of lcov.
2018-09-15 13:27:06 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
cf889790e6 Error if LibC build is performed outside test environment.
LibC is no longer required for production builds.
2018-07-10 15:39:03 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
a385cb520b Update primary test environment (Vagrant and Docker) to Ubuntu 18.04. 2018-06-06 15:52:28 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
8eb9921c7c Auto-generate Makefile with dependencies. 2018-05-23 14:57:08 -04:00
David Steele
f7123c1bbf v2.03: Single Executable to Deploy
Bug Fixes:

* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)

Improvements:

* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c Test harness improvements.
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
790f7c7312 Enforce full branch coverage in C code. 2018-05-05 11:50:11 -04:00
David Steele
ea4046a607 Try tweaking time sync settings to prevent clock drift rather than restarting VBoxService on every test run. 2018-04-18 18:47:43 -04:00
David Steele
dce11ce3cf Generate code counts for all source files.
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00