Additional Notesbut changes in this section are only to documentation or the test suite and have no direct impact the on the
IMPORTANT NOTE: The
Fixed const assignment giving compiler warning in C library.
Fixed a few directory syncs that were missed for the
Fixed an issue where a missing user/group on restore could cause an uninitialized value
error in File->owner()
.
Improved
Improved pg_switch_xlog()
cannot be executed on a replica.
Refactor File
and BackupCommon
modules to improve test coverage.
Return proper error code when unable to convert a relative path to an absolute path.
Added documentation to the User Guide for the
Update LICENSE.txt for 2017.
Added unit tests for low-level functions in the File
and BackupCommon
modules.
Split test modules into separate files to make the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
Allow multiple
Added expect log expression to replace year subdirectories in
IMPORTANT NOTE: In prior releases it was possible to specify options on the command-line that were invalid for the current command without getting an error. An error will now be generated for invalid options so it is important to carefully check command-line options in your environment to prevent disruption.
Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error. The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error.
Fixed an issue where internal symlinks were not being created for tablespaces in the repository. This issue was only apparent when trying to bring up clusters in-place manually using filesystem snapshots and did not affect normal backup and restore.
Fixed an issue that prevented errors from being output to the console before the logging system was initialized, i.e. while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not.
Fixed an issue where the
Added the
Added the
Added the
Added a predictable log entry to signal that a command has completed successfully. For example a backup ends successfully with: INFO: backup command end: completed successfully
.
Abstracted code to determine which database cluster is the master and which are standbys.
Improved consistency and flexibility of the protocol layer by using JSON for all messages.
File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process.
Improved IO->bufferRead
to always return requested number of bytes until EOF.
For simplicity, the
Simplified the result hash of File->manifest()
, Db->tablespaceMapGet()
, and Db->databaseMapGet()
.
Improved errors returned from child processes by removing redundant error level and code.
Code cleanup in preparation for improved
Improved parameter/result logging in debug/trace functions.
Fixed an issue that suppressed exceptions in PDF builds.
Fixed regression in section links introduced in
Added Retention to QuickStart section.
Allow a source to be included as a section so large documents can be broken up.
Added section link support to Markdown output.
Added list support to PDF output.
Added
Added
Added
Added
Builds in
Improvements to markdown rendering.
Remove code dependency on
Removed erroneous
Update control and WAL test files to
Improved exception handling in file unit tests.
Changed the
Various improvements to validation of backup and restore.
Add more realistic data files to synthetic backup and restore tests.
Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches. This regression was introduced in
Fixed an issue where a backup could error if no changes were made to a database between backups and only
Fixed an issue where tablespace paths with the same prefix would cause an invalid link error.
Added the
Removed extraneous use lib
directives from Perl modules.
Fixed missing variable replacements.
Removed hard-coded host names from configuration file paths.
Allow command-line length to be configured using
Added
Added
Only generate HTML menu when required and don't require index page.
Assign numbers to sections by default.
VM mount points are now optional.
Fixed the
Fixed error message to properly display the archive command when an invalid archive command is detected.
Fixed an issue where the async archiver would not be started if
Fixed a remote timeout that occurred when a local process generated checksums (during resume or restore) but did not copy files, allowing the remote to go idle.
Non-exclusive backups will automatically be used on
Added the
Added the
Set
Check that archive_mode is enabled when
Clarified error message when unable to acquire
Process ID logged for local process start/stop INFO log output.
Added
Added
Fixed missing expect output for help module.
Fixed broken
Update CentOS/Debian package definitions.
Regression tests can now be run as any properly-configured user, not just vagrant.
Miminize TeXLive package list to save time during VM builds.
Fixed an issue where local processes were not disconnecting when complete and could later timeout.
Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive.
Cache file log output until the file is created to create a more complete log.
Show Process ID in output instead of filtering it out with the timestamp.
Suppress dpkg-reconfigure: unable to re-open stdin: No file or directory
warning in Vagrant VM build.
Show Process ID in expect logs instead of filtering it out with the timestamp.
Fixed an issue where tablespaces were copied from the master during standby backup.
Fixed the
Fixed an issue where
Converted Perl threads to processes to improve compatibility and performance.
Exclude contents of
The
Additional warnings when archive retention settings may not have the intended effect or would allow indefinite retention.
Experimental support for non-exclusive backups in
Refactor of protocol minions in preparation for the new local minion.
Remove obsolete thread index variable from File()
module.
Changed temporary file names to consistently use the
Improve ASSERT error handling, safely check eval blocks, and convert $@
to $EVAL_ERROR
.
Fixed minor documentation reproducibility issues related to binary paths.
Documentation for archive retention.
Suppress TOC for unsupported versions of
New vagrant base box and make uid/gid selection for containers dynamic.
Fixed an issue where a tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely.
Fixed an issue where options that should not allow multiple values could be specified multiple times in
Fixed an issue where the
Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
More flexible configuration for databases. Master and standby can both be configured on the backup server and
Exclude directories during backup that are cleaned, recreated, or zeroed by
Experimental support for non-exclusive backups in
Simplify protocol creation and identifying which host is local/remote.
Removed all OP_*
function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__
.
Improvements in Db
module: separated out connect()
function, allow executeSql()
calls that do not return data, and improve error handling.
Improve error message for links that reference links in manifest build.
Added hints to error message when relative paths are detected in
Improve backup log messages to indicate which host the files are being copied from.
Improve host tag rendering.
Refactor db version constants into a separate module.
Update synthetic backup tests to
Fixed an issue where tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not. Also hardened relative path checking a bit.
Added documentation for scheduling backups with cron.
Moved the backlog from the
Improved rendering of spaces in code blocks.
Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict.
Fixed an issue where db-path was not required for the
Fixed
Fixed an issue where a remote could try to start its own remote when the
Fixed an issue where the contents of
Fixed an invalid log()
call in lock routines.
Experimental support for non-exclusive backups in
Enhancements to the protocol layer for improved reliability and error handling.
All remote types now take locks. The exceptions date to when the test harness and
Exceptions are now passed back from threads as messages when possible rather than raised directly.
Temp files created during backup are now placed in the same directory as the target file.
Output lock file name when a lock cannot be acquired to aid in debugging.
Reduce calls to protocolGet()
in backup/restore.
Suppress banners on SSH protocol connections.
Improved remote error messages to identify the host where the error was raised.
Added
Added clarification on why the default for the
Updated information about package availability on supported platforms.
HTML footer dates are statically created in English in order to be reproducible.
Fixed a version checking issue in
Fixed an issue where multi-threaded tests were not being run when requested.
Reduce the frequency that certain tests are run to save time in regression.
Disable control master for older OS versions where it is less stable.
Fixed an issue where
Fixed an issue where the
Fixed an issue where validation was being running on
Added
Added the
Failure to shutdown remotes at the end of the backup no longer throws an exception. Instead a warning is generated that recommends a higher
Experimental support for non-exclusive backups in
The pg_xlogfile_name()
function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the
Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
Changed version variable to a constant. It had originally been designed to play nice with a specific packaging tool but that tool was never used.
Fixed DTD search path that did not work properly when
Fixed
Fixed section names being repeated in the info output when multiple
Fixed
Allow hidden options to be added to a command. This allows certain commands (like
Allow command summaries to be inserted anywhere in the documentation to avoid duplication.
Allow a static date to be used for documentation to generate reproducible builds.
Update TeX Live to 2016 version.
Added documentation for asynchronous archiving to the user guide.
Recommended install location for
Added instructions for removing prior versions of
New, consolidated implementation for link rendering.
Obsolete containers are removed by the
Major refactor of the test suite to make it more modular and object-oriented. Multiple Docker containers can now be created for a single test to simulate more realistic environments. Tests paths have been renamed for clarity.
Greatly reduced the quantity of Docker containers built by default. Containers are only built for
Fix usage of sprintf() due to new constraints in Perl 5.22. Parameters not referenced in the format string are no longer allowed.
Log directory create and file open now using FileCommon functions which produce more detailed error messages on failure.
Fixed syntax that was not compatible with Perl 5.2X.
Fixed absolute paths that were used for the PDF logo.
Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated. An Additional Notes
section has been added for changes to documentation and the test suite that do not affect the core code.
Added man page generation.
Added an execution cache so that documentation can be generated without setting up the full container environment. This is useful for packaging, keeps the documentation consistent for a release, and speeds up generation when no changes are made in the execution list.
The change log was the last piece of documentation to be rendered in Markdown only. Wrote a converter so the document can be output by the standard renderers. The change log will now be located on the website and has been renamed to Releases
.
Remove function constants and pass strings directly to logDebugParam(). The function names were only used once so creating constants for them was wasteful.
Lists can now be used outside of
Replaced overzealous perl -cW
check which failed on Perl 5.22 with perl -cw
.
Added Ubuntu 16.04 (Xenial) and Debian 8 (Jessie) to the regression suite.
Upgraded doc/test VM to Ubuntu 16.04. This will help catch Perl errors in the doc code since it is not run across multiple distributions like the core and test code. It is also to be hoped that a newer kernel will make Docker more stable.
Test release version against the executable using
Enhanced text output of
Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
Experimental support for non-exclusive backups in
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of
Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the
The
The default configuration filename is now
The default repository name was changed from
Lock files are now stored in
Log files are now stored in
Executable filename changed from
All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The
Removed
Added
Fixed an issue where the master process was passing
IMPORTANT BUG FIX FOR TABLESPACES: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid
Fixed repository incompatibility introduced in
Copy
Write
Rename
Static source analysis using Perl-Critic, currently passes on gentle.
Fixed an issue where specifying
Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next
The
When backing up and restoring tablespaces
Generate an error when
Improved error message when
Added checks for
When restore
Support for
Support for
The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors.
Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of
Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier.
Allow restores to be performed on a read-only repository by using
Minor styling changes, clarifications and rewording in the user guide.
The dev branch has been renamed to master and for the time being the master branch has renamed to release, though it will probably be removed at some point {[dash]}- thus ends the gitflow experiment for
Fixed an issue where the
Fixed an issue where log files were being overwritten instead of appended.
Fixed an issue where
Symlinks are no longer created in backup directories in the repository. These symlinks could point virtually anywhere and potentially be dangerous. Symlinks are still recreated during a restore.
Added better messaging for backup expiration. Full and differential backup expirations are logged on a single line along with a list of all dependent backups expired.
Archive retention is automatically set to full backup retention if not explicitly configured.
Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping
The
Added a new user guide that covers
The website, markdown, and command-line help are now all generated from the same XML source.
Fixed an issue where an error could be returned after a backup or restore completely successfully.
Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.
Added
Experimental support for
Removed dependency on IO::String
module.
Fixed an issue where resumed compressed backups were not preserving existing files.
Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.
Fixed an issue where a cold backup using
Fixed an issue where a thread could be started even when none were requested.
Fixed an issue where the
Fixed an issue where the
Fixed an issue where the pg_stop_backup()
NOTICEs were being output to
Experimental support for
Renamed
Code cleanup and refactoring to standardize on patterns that have evolved over time.
Added dynamic module loading to speed up commands, especially asynchronous archiving.
Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.
Expiration tests are now synthetic rather than based on actual backups. This will allow development of more advanced expiration features.
Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the
Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.
The repository is now created and updated with consistent directory and file modes. By default
Added the
Added the pg_start_backup()
and pg_stop_backup()
to return.
Remove
Added checks to be sure the
Experimental support for
Now using Perl DBI
and DBD::Pg
for connections to DBD::Pg
on your operating system.
Major refactoring of the protocol layer to support future development.
Split most of database
in the user guide to database cluster
for clarity.
Changed most references to database
in the user guide to database cluster
for clarity.
Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.
Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the threads
and Thread::Queue
packages, it is no longer necessary.
Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.
Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.
More options for regression tests and improved code to run in a variety of environments.
Added file and directory syncs to the File
object for additional safety during backup/restore and archiving.
Added support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6.
Improved error message when backup is run without
Removed
Moved version number out of the
Replaced IPC::System::Simple
and Net::OpenSSH
with IPC::Open3
to eliminate CPAN dependency for multiple operating systems.
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of
Added the
Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic.
Experimental support for
More efficient file ordering for
Fixed an issue where
Fixed an issue in async archiving where
Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore.
Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.
Allow comments/disabling in
Better logging before
Various command behavior and logging fixes.
Replaced JSON
module with JSON::PP
which ships with core Perl.
Various help fixes.
Fixed an issue where an absolute path was not written into
Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.
Resume is now optional. Use the
More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.
Added
Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.
Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.
Improved threading model by starting threads early and terminating them late.
Regression tests working back to
Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.
Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
Added restore functionality.
All options can now be set on the command-line making
De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
Added option
Checksum for
Link
Removed dependency on Moose
. It wasn't being used extensively and makes for longer startup times.
More comprehensive unit tests in all areas.
Complete rewrite of BackRest::File
module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.
Removed dependency on Storable
and replaced with a custom ini file implementation.
Numerous other changes that can only be identified with a diff.
Added much needed documentation
Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
Found and squashed a nasty bug where file_copy()
was defaulted to ignore errors. There was also an issue in file_exists()
that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
Worked on improving error handling in the File
object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).
The
If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
Added
Added
Added option to force a checkpoint when starting the backup,
Removed
Changed lock file conflicts on
No restore functionality, but the backup directories are consistent
Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
Backups are multi-threaded, but the Net::OpenSSH
library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
The Storable
because Config::IniFile
does not seem to handle large files well. Would definitely like to save these as human-readable text.
Absolutely no documentation (outside the code). Well, excepting these release notes.