1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-05-13 21:56:42 +02:00

46 Commits

Author SHA1 Message Date
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
e4408c7dd3 Refactor --pre option in documentation.
Mark all pre commands as skip so they won't be run again after the container is built.

Ensure that pre commands added to the container are run as the container user if they are not intended to run as root.
2019-10-11 12:28:47 -04:00
David Steele
51fcaee43e Add host-repo-path variable internal replacement.
This variable needs to be replaced right before being used without being added to the cache since the host repo path will vary from system to system.

This is frankly a bit of a hack to get the documentation to build in the Debian packages for the upcoming release.  We'll need to come up with something more flexible going forward.
2019-06-25 07:58:38 -04:00
David Steele
3e1b06acaa Use minio as local S3 emulator in documentation.
The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.

Also, install certificates and CA so that TLS verification can be enabled.
2019-05-27 07:37:20 -04:00
David Steele
17e611cb88 Replace keywords with more flexible if statements.
Keywords were extremely limited and prevented us from generating multi-version documentation and other improvements.

Replace keywords with an if statement that can evaluate a Perl expression with variable replacement.

Since keywords were used to generate cache keys, add a --key-var parameter to identify which variables should make up the key.
2018-12-01 12:40:01 -05:00
David Steele
5d3c8e47f1 Pre-build containers for any execute elements marked pre.
This allows the documentation to be built more quickly and offline during development when --pre is specified on the command line.

Each host gets a pre-built container with all the execute elements marked pre. As long as the pre elements do not change the container will not need to be rebuilt.

The feature should not be used for CI builds as it may hide errors in the documentation.
2018-11-29 14:45:15 -05:00
David Steele
78fe642eae Remove extraneous use/include statements.
Use conditional loading to make docs work in the absence of LibC.

Somehow this also required a use statement to be added. Perl, go figure.
2018-11-24 20:31:35 -05:00
David Steele
801e2a5a2c Rename PGBACKREST/BACKREST constants to PROJECT.
This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-24 19:05:03 -05:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
e57840d739 Document generator improvements.
* Allow parameters to be passed when a container is created.
* Allow /etc/hosts update to be skipped (for containers without bash).
2018-05-25 13:42:09 -04:00
David Steele
5eb682a569 Add id param for hosts created with host-add.
The host-*-ip variable is created from the id param so the name param can be changed without affecting the host-*-ip variable. If id is not specified then it is copied from name.
2018-02-21 18:10:53 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
David Steele
7cf955425e The C library is now required.
This eliminates conditional loading and eases development of new library features.
2017-11-26 17:45:00 -05:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
97c4d9ec18 Reduce log verbosity in doc.pl.
Reduce verbosity by only logging sections that contain an execute list directly or in a child section.
2017-07-26 10:22:22 -04:00
David Steele
884413b251 Remove mount and user from host cache-key because it can vary by system. 2017-06-27 22:27:48 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
David Steele
5c635e0f0a Go back to using static user for documentation.
Making this dynamic in commit 5d2e792 broke doc builds from cache.  The long-term solution is to create a special user for doc builds but that’s beyond the scope of this release.
2017-04-12 18:36:33 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
5d2e79242d Fixed user change in docs missed in vagrant upgrade. 2017-03-16 19:50:59 -04:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
498f52da09 Improvements to documentation engine:
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
2017-02-10 10:22:05 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
9d9c438f59 Doc engine improvements.
Bug Fixes:

* Fixed missing variable replacements.
* Removed hard-coded host names from configuration file paths.

Documentation Features:

* Allow command-line length to be configured using cmd-line-len param.
* Added compact param to allow CSS to be embedded in HTML file.
* Added pretty param to produce HTML with proper indenting.
* Only generate HTML menu when required and don't require index page.
* Assign numbers to sections by default.
* VM mount points are now optional.
2016-11-07 17:06:35 +02:00
David Steele
2a18674111 Added the log-level-stderr option.
Controls whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior.

Suggested by Sascha Biberhofer.
2016-10-05 09:09:30 -04:00
David Steele
e08b56a4e3 Show Process ID in log output instead of filtering it out with the timestamp. 2016-09-14 16:30:57 -05:00
David Steele
dd8d781217 Safely check eval blocks and convert $@ to $EVAL_ERROR. 2016-09-06 09:44:50 -04:00
David Steele
a701309453 Converted Perl threads to processes. 2016-09-06 09:35:02 -04:00
David Steele
d0b6f78b20 More flexible configuration for databases
Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 12:39:27 -04:00
David Steele
c8f806a293 Major refactor of the test suite.
* Make the code more modular and object-oriented.
* Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 08:12:58 -04:00
David Steele
4cff5bb0e6 Fixed pgBackRest config sections being blank in the output when not loaded from cache. 2016-06-18 10:59:45 -04:00
David Steele
e796f563c4 Add cmd-extra field.
Allow hidden options to be added to a command. This allows certain commands (like apt-get) to be forced during the build without making that a part of the documentation.
2016-06-03 20:07:28 -04:00
David Steele
eaa981c5aa Added execution cache for document generation.
Added an execution cache so that documentation can be generated without setting up the full container environment. This is useful for packaging, keeps the documentation consistent for a release, and speeds up generation when no changes are made in the execution list.
2016-06-02 09:32:56 -04:00
David Steele
c8d68bcf2d More detailed release notes.
Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated.  An "Additional Notes" section has been added for changes to documentation and the test suite that do not affect the core code.
2016-05-26 10:34:10 -04:00
Cynthia Shang
5a85122841 Moved change log to website.
The change log was the last piece of documentation to be rendered in Markdown only.  Wrote a converter so the document can be output by the standard renderers.  The change log will now be located on the website and has been renamed to "Releases".
2016-05-26 09:20:55 -04:00
David Steele
e2094c3d31 Remove function constants and pass strings directly to logDebugParam().
The function names were only used once so creating constants for them was wasteful.
2016-05-26 09:09:42 -04:00
Christoph Berg
3a309256fa Closed #193: Fix perl warnings in doc/ tree.
Somewhere between perl 5.14 and 5.20, constructs like this:
    perl -e '$a = {}; keys $a'
started to throw warnings:
    keys on reference is experimental at -e line 1.

Fix by adding a bunch of %{} and @{} casts.
2016-05-19 10:25:04 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00
David Steele
723abadb75 Change spacing in expect logs to accommodate DETAIL level. 2016-04-13 18:55:34 -04:00
David Steele
7f630aec49 Switched to using docker for all unit tests as this allows for greater automation. 2015-12-29 13:57:10 -05:00
David Steele
fa05715dec Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication. 2015-11-22 16:44:01 -05:00