1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

278 Commits

Author SHA1 Message Date
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process.
Fixed by Cynthia Shang.
2017-05-16 11:28:51 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages.
Contributed by Cynthia Shang.
2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers.
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
2017-04-10 12:31:30 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade.
Contributed by Cynthia Shang.
2017-04-06 19:36:37 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash.
Refactor File->list() and fileList() to accept optional parameters.
2017-04-04 21:17:19 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
Contributed by Cynthia Shang.
2017-03-22 19:03:28 -04:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
023578c008 Make package tests work with dev builds. 2017-02-21 14:22:03 -05:00
David Steele
0e9bc513c9 Suppress lint on all but the first test. 2017-02-21 12:54:27 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest().
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-02-21 11:56:55 -05:00
David Steele
01312f6260 Improved formatting of testResult() output. 2017-02-21 11:47:45 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
56543f7a99 Added package builds to test suite and other improvements:
* Automated builds of Debian packages for all supported distributions.
* Added --dev option to aggregate commonly used dev options.
* Added --no-package option to skip package builds.
* C library and packages are built by default, added -smart option to rebuild only when file changes are detected.
* The --libc-only option has been changed to --build-only now that packages builds have been added.
2017-02-17 22:31:16 -05:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
498f52da09 Improvements to documentation engine:
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
2017-02-10 10:22:05 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
18db178ac3 Various improvements to the test suite:
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
2017-01-27 09:42:30 -05:00
David Steele
0e4f51c271 Added integration for testing coverage with Devel::Cover. 2017-01-09 20:49:04 -05:00
David Steele
39744a4f1d Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner().
Reported by Leonardo Avellar.
2017-01-05 15:19:56 -05:00
David Steele
4ff2714bf0 Return proper error code when unable to convert a relative path to an absolute path.
Suggested by Yogesh Sharma.
2017-01-04 18:39:50 -05:00
David Steele
c8dfc673b8 Moved tests into new modules to better reflect actual usage. 2017-01-04 18:31:16 -05:00
David Steele
e3d63c0252 Allow multiple --module, --test, and --run options to be used for test.pl. 2017-01-04 18:29:13 -05:00
Cynthia Shang
be337c951a Improvements to stanza-create:
* Force only required when a change is needed for backup.info or archive.info.
* Identical files are not overwritten.
2017-01-04 10:12:29 -05:00
David Steele
b91ca11329 Added expect log expression to replace year subdirectories in backup.history. 2017-01-04 10:09:21 -05:00
David Steele
83beab7ec3 Refactor File module to improve test coverage.
Added unit tests for low-level functions in the BackupCommon module.
2016-12-23 11:43:26 -05:00
David Steele
5d3473b52d Refactor File module to improve test coverage. 2016-12-23 08:30:34 -05:00
David Steele
6b2666a9d7 Split test modules into separate files.
Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 08:22:59 -05:00
Cynthia Shang
974a02dbe4 Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory.
Contributed by Cynthia Shang.
2016-12-20 16:52:20 -05:00
David Steele
36a5349b1c Added the --checksum-page option.
This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-12 18:54:07 -05:00
David Steele
b4884e5a0b Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database.
Reported by Michael Vitale.
2016-12-10 14:05:28 -05:00
David Steele
eff7b46eb1 Various improvements to validation of backup and restore. 2016-12-10 09:11:12 -05:00
David Steele
f818901c0e Added the --repo-sync option to allow directory syncs in the repository to be disabled.
This is for file systems that do not support them, e.g. NTFS
2016-12-10 09:06:45 -05:00
David Steele
817592837c Added a predictable log entry to signal that a command has completed successfully.
For example a backup ends successfully with: INFO: backup command end: completed successfully.
2016-12-04 14:44:53 -05:00
David Steele
d18cd7ad0f Fixed an issue that prevented errors from being output to the console before the logging system was initialized.
That is, while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not.
2016-12-03 18:01:17 -05:00
David Steele
2e01e6b9e1 Improved consistency and flexibility of the protocol layer by using JSON for all messages. 2016-12-03 17:34:51 -05:00
David Steele
9d9c438f59 Doc engine improvements.
Bug Fixes:

* Fixed missing variable replacements.
* Removed hard-coded host names from configuration file paths.

Documentation Features:

* Allow command-line length to be configured using cmd-line-len param.
* Added compact param to allow CSS to be embedded in HTML file.
* Added pretty param to produce HTML with proper indenting.
* Only generate HTML menu when required and don't require index page.
* Assign numbers to sections by default.
* VM mount points are now optional.
2016-11-07 17:06:35 +02:00
David Steele
1491862716 Regression tests can now be run as any properly-configured user, not just vagrant. 2016-10-01 13:39:44 -04:00
David Steele
156859c5d5 Update CentOS/Debian package definitions. 2016-09-29 19:07:46 -04:00
David Steele
25f40f9b15 Log improvements:
* pgBackRest version number included in command start INFO log output.
* Process ID logged for local process start/stop INFO log output.
* Fixed missing expect output for help module.
2016-09-17 11:53:28 -04:00
David Steele
f54145c0cc Fixed timeout issues.
* Fixed an issue where local processes were not disconnecting when complete and could later timeout. (Reported by Todd Vernick.)
* Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive. (Reported by Todd Vernick.)
2016-09-14 16:37:07 -05:00
David Steele
e08b56a4e3 Show Process ID in log output instead of filtering it out with the timestamp. 2016-09-14 16:30:57 -05:00
David Steele
cc9c8819ee Make uid/gid selection for containers dynamic. 2016-09-06 14:01:39 -04:00
David Steele
dd8d781217 Safely check eval blocks and convert $@ to $EVAL_ERROR. 2016-09-06 09:44:50 -04:00
David Steele
a701309453 Converted Perl threads to processes. 2016-09-06 09:35:02 -04:00
David Steele
d0b6f78b20 More flexible configuration for databases
Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 12:39:27 -04:00
David Steele
17b79d6279 Database version refactoring.
* Refactor db version constants into a separate module.
* Update synthetic backup tests to PostgreSQL 9.4.
2016-08-11 22:35:24 -04:00
David Steele
bff262ac47 Removed all OP_* function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__. 2016-08-11 17:32:28 -04:00
David Steele
94a4cb4123 Disable control master for older OS versions where it is less stable. 2016-07-30 08:43:38 -04:00
David Steele
273d1c08d8 Improved remote error messages to identify the host where the error was raised. 2016-07-26 16:57:38 -04:00
David Steele
b6ac6b0f1c Suppress banners on SSH protocol connections. 2016-07-26 16:28:43 -04:00
David Steele
5df2eed729 Order database versions during test.pl --vm-build. 2016-07-02 09:07:48 -04:00
David Steele
bd25223fd6 Rename test paths for clarity.
This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved.  Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 10:54:31 -04:00
David Steele
c8f806a293 Major refactor of the test suite.
* Make the code more modular and object-oriented.
* Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 08:12:58 -04:00
David Steele
f1e92804ce Docker container improvements.
* Containers now use a squid proxy for apt/yum to speed builds.
* Obsolete containers are removed by the <br-option>--vm-force</br-option> option.
* Greatly reduced the quantity of Docker containers built by default. Containers are only built for PostgreSQL versions specified in db-minimal and those required to build documentation. Additional containers can be built with --db-version=all or by specifying a version, e.g. --db-version=9.4.
2016-06-12 09:00:16 -04:00
David Steele
e7d4a25d6a Added Ubuntu 16.04 and Debian 8 to regression tests.
Also added some new fields to make container generation simpler for distros that share a common package base.
2016-05-24 08:17:13 -04:00
David Steele
0451d3afdd Support for non-exclusive backups in PostgreSQL 9.6. 2016-05-16 17:59:26 -04:00
David Steele
512d006346 Refactor database version identification for archive and backup commands.
Added database version constants and changed version identification code to use hash tables instead of if-else.  Propagated the db version constants to the rest of the code and in passing fixed some path/filename constants.

Added new regression tests to check that specific files are never copied.
2016-05-14 10:33:12 -04:00
David Steele
a6a19e3735 Test directories are now located on the host VM rather than in the Docker container.
This change allows for easier testing since all files are local on the host VM and can be easily accessed without using `docker exec`.  In addition, this change is required to allow multiple Docker containers per test case which is coming soon.
2016-05-10 18:12:37 -04:00
Cynthia Shang
60b901948a Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. 2016-05-06 10:25:46 -04:00
David Steele
9457e15347 New manifest format.
* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 22:50:02 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00