Add most unimplemented functions to the remote storage driver.

Filter improvements. Only process next filter in IoFilterGroup when input buffer is full or flushing. Improve filter's notion of done to optimize filter processing.

Improve performance of non-blocking reads by using maximum buffer size.

Add storageInfoList() to get detailed info about all entries in a path.

Allow storageInfo() to follow links.

Allow StorageFileWrite to set user, group, and modification time.

Add pathExists() to Storage object.

Improve zero-length content handling in HttpClient object.

Don't append strerror() to error message when errno is 0.

Improve type safety of interfaces and drivers.

Add --c option to request a C remote.

Add common/macro.h for general-purpose macros.

Add macros for object free functions.

Various MemContext callback improvements.

Various Buffer improvements.

Simplify storage object names.

Add ioWriteStr() and ioWriteStrLine().

Add separate functions to encode/decode each JSON type.

Add macros to create constant Buffer objects.

Add missing httpUriEncode() in S3 request.

Add unsigned int Variant type and update code to use it.

Expose handle (file descriptor) from IoWrite when applicable.

Add iniSave() and iniMove() to Ini object.

Add *Save() functions to most Info objects.

Extern infoHash() so it can be used by other modules.

varNewKv() accepts a KeyValue object rather than creating one.

Add constant for maximum buffer sizes required by cvt*() functions.

Add true and false String constants.

Refactor Ini interface to expose String values instead of Variant.

Refactor main() as a switch() statement.

Add cfgOptionUInt() and cfgOptionUInt64() and update code to use them.

Improve log performance, simplify macros, rename logWill() to logAny().

Improve coverage in perl/exec, config/config, and config/parse modules.

Remove -Wswitch-enum compiler option.

Error on multiple option alternate names and simplify help command.

Use THROW_ON_SYS_ERROR*() to improve code coverage.

Improve macros and coverage rules that were hiding missing coverage.

Improve efficiency of FUNCTION_LOG*() macros.

Harden IO filters against zero input and optimize zero output case.

Move lockRelease() to the end of exitSafe().

Add CHECK() macro for production assertions.

Automatically generate constants for command and option names.

Use a macro instead of a nested struct to create common String variables.

Add STR() macro to create constant String objects from runtime strings.

Add macros to create constant Variant types.

Migrate backupRegExp() to C.

Add separate archive-push-async command.

CryptoHash improvements and fixes. Fix incorrect buffer size used in cryptoHashOne(). Add missing const to cryptoHashOne() and cryptoHashOneStr(). Add hash size constants. Extern hash type constant.

Add CIFS storage driver.

Add file write to the remote and S3 storage drivers.

Add storageRepoWrite() and storagePg()/storagePgWrite() to storage helper.

Use a single file to handle global errors in async archiving.

Add document creation to XML objects.

Remove redundant documentation from interface files and clarify ambiguous function names.

Add WAL info to interface.

Refactor interface to remove most code duplication.

Logging improvements. Allow three-digit process IDs in logging. Allow process id in C logging.

Add process id to ProtocolParallelJob.

Add process id to C archive-get and archive-push logging.

Close log file before exec().

Allow warnings to be written by archiveAsyncStatusOkWrite().

Move WAL path prefix logic into walPath().

Make notion of current info ID in C align with Perl.

Add locking capability to the remote command.

Add forkSafe() to handle fork errors.

Add httpHeaderDup().

httpClientRequest() accepts a body parameter.

Add protocolKeepAlive() to send noops to all remotes.

Make strLstDup() null-tolerant.

Add strLstMergeAnti() for merge anti-joins.

Add cvtSSizeToZ() and debug macros.

Remove unused infoArchiveCheckPg() function.

Add constants for .ok/.error status extensions.

Migrate local and remote commands to C.

Add separate archive-get-async command.

Add ProtocolParallel* objects for parallelizing commands.

Add ProtocolCommand object.

Add exists() to remote storage.

Resolve storage path expressions before passing to remote.

Expose handle (file descriptor) from IoRead when applicable.

Add storageHelperFree() to storage helper.

Add kvKeyExists() to KeyValue object.

Add lstRemove() to List object.

Allow cfgExecParam() to exclude options.

MemContext improvements. Improve performance of context and memory allocations. Use contextTop/contextCurrent instead of memContextTop()/memContextCurrent(). Don't make a copy of the context name.

Make DESTDIR fully-configurable in the Makefile.

Add note for CSTD settings on BSD variants.

Add clean and uninstall targets to Makefile.

Change execRead() to return a size_t.

Prevent option warning from being output when running help command.

Improve null-handling of strToLog() and varToLog().

Increase per-call stack trace size to 4096.

Move compress module to common/compress.

Move crypto module to common/crypto.

JSON improvements. Optimize parser implementation. Make the renderer more null tolerant.

Automatically adjust db-timeout when protocol-timeout is smaller.

Add remote storage objects.

Add ProtocolClient object and helper functions.

Add Exec object.

Add IoHandleRead and IoHandleWrite objects.

Add cfgExecParam() to generate parameters for executing commands.

Ignore SIGPIPE signals and check EPIPE result instead.

Function log macro improvements. Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Improve CONST and P/PP type macro handling. Move MACRO_TO_STR() to common/debug.h. Remove unused type parameter from FUNCTION_TEST_RETURN().

Make the C version of the info command conform to the Perl version.

Improve accuracy of strSizeFormat().

Add ioReadBuf() to easily read into a buffer.

JSON improvements. Allow empty arrays in JSON parser. Fix null output in JSON renderer. Fix escaping in JSON string parser/renderer.

Allocate extra space for concatenations in the String object.

Return UnknownError from errorTypeFromCode() for invalid error codes.

Update Perl repo rules to work when stanza is not specified.

Update Storage::Local->list() to accept an undefined path.

Null-terminate list returned by strLstPtr().

Add kvMove() and varLstMove().

Replace FileOpenError with HostConnectError in TlsClient.

Allow string Variant objects to contain null.

Rename common/io/handle module to common/io/handleWrite.

Add const VariantList * debug type.

Enable S3 storage and encryption for archive-get command in C.

Migrate local info command to C.

Add S3 storage driver.

Add HttpClient object.

Add TlsClient object.

Add interface objects for libxml2.

Add encryption capability to Info* objects.

Add IoFilter interface to CipherBlock object.

Allow arbitary InOut filters to be chained in IoFilterGroup.

Add infoBackup object to encapsulate the backup.info file.

Improve JSON to Variant conversion and add Variant to JSON conversion.

Storage helper improvements. Allow NULL stanza in storage helper. Add path expression for repository backup.

Info module improvements. Rename constants in Info module for consistency. Remove #define statements in the InfoPg module to conform with newly-adopted coding standards. Use cast to make for loop more readable in InfoPg module. Add infoArchiveIdHistoryMatch() to the InfoArchive object.

Allow I/O read interface to explicitly request blocking reads.

Improve error messages when info files are missing/corrupt.

Add strSizeFormat() to String object.

Add strLstInsert() and strLstInsertZ() to StringList object.

Rename PGBACKREST/BACKREST constants to PROJECT.

Require S3 key options except for local/remote commands.

Explicitly compile with Posix 2001 standard.

Add ServiceError for errors from a service that can be retried.

Conditional compilation of Perl logic in exit.c.

Merge cipher.h into crypto.h.

Remove extraneous use/include statements.

Remove embedded semicolon from String constant macros.

Reduce debug level for infoIni() to test.

Return IoFilterGroup * from ioFilterGroupAdd().

Add coding standards for String constants.

Add missing LOG_DEBUG() macro.

Correct current history item in InfoPg to always be in position 0.

Make ioReadLine() read less aggressively.

Add ioWriteFlush() to flush pending output.

Add destructors to IoRead and IoWrite objects.

Add base variants to all integer to string conversion functions.

Add lstInsert() to List object.

Add strCatChr(), strEmpty(), and constant macros to String object.

Add regExpPrefix() to aid in static prefix searches.

Correct cfgDefDataFind() to use UINTP instead of VOIDPP.

Change infoArchiveCheckPg() to display the version as a string (e.g. 9.4) instead of the integer representation (e.g. 90400) when throwing an error.

Allow storage path and file mode to be 0.

Limit usable Buffer size without changing allocated size.

Construct Wait object in milliseconds instead of fractional seconds.

Add THROW*_ON_SYS_ERROR* macros to test and throw system errors.

Add KernelError to report miscellaneous kernel errors.

Use THROW_ON_SYS_ERROR macro to improve fork code coverage.

Storage interface methods no longer declare the driver as const.

Add memContextCallbackClear() to prevent double free() calls.

Merge crypto/random module into crypto/crypto.

Add cryptoError() and update crypto code to use it.

Rename CipherError to CryptoError.

Reword misleading message in stack trace when parameter buffer is full.

Add logging macros for TimeMSec type.

Modify value of PERL_EMBED_ERROR macro.

Migrate local, unencrypted, non-S3 archive-get command to C.

Storage refactoring. Posix file functions now differentiate between open and missing errors. Don't use negations in objects below Storage. Rename posix driver files/functions for consistency. Full abstraction of storage driver interface. Merge protocol storage helper into storage helper. Add CIFS driver to storage helper for read-only repositories.

Update all interfaces to use variable parameter constructors.

Info objects now parse JSON and use specified storage.

Add ioReadLine()/ioWriteLine() to IoRead/IoWrite objects.

Add helper for repository storage.

Add cryptoHmacOne() for HMAC support.

Add cfgDefOptionMulti() to identify multi-value options.

Add bufNewZ() and bufHex() to Buffer object.

Allow hashSize() to run on remote storage.

Restore bIgnoreMissing flag in backupFile() lost in storage refactor.

Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C.

Migrate control functions to detect stop files to C.

Make archive-get info messages consistent between C and Perl implementations.

Change locking around async process forking to be more test friendly.

Simplify debug logging by allowing log functions to return String objects.

Improve documentation in filter.h and filter.internal.h.

Validate configuration options in a single pass. By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.

Add gzip compression/decompression filters for C.

Improve performance of string to int conversion. Use strtoll() instead of sprintf() for conversion. Also use available integer min/max constants rather than hard-coded values.

Add uint64 variant type and supporting conversion functions.

Add basic C JSON parser.

Migrate minimum set of code for reading archive.info files from Perl to C.

Allow Buffer object used size to be different than allocated size. Add functions to manage used size and remaining size and update automatically when possible.

Abstract IO layer out of the storage layer. This allows the routines to be used for IO objects that do not have a storage representation. Implement buffer read and write IO objects. Implement filters and update cryptoHash to use the new interface. Implement size and buffer filters.

storageFileRead() accepts a buffer for output rather than creating one. This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Add iniSectionList() to Ini object and remove dead code.

Manifest improvements. Require catalog version when instantiating a Manifest object (and not loading it from disk). Prevent manifest from being built more than once. Limit manifest build recursion (i.e. links followed) to sixteen levels to detect link loops.

Do nothing in memContextMove() when the context is already in the specified parent.

Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added.

Show exact log level required for stack trace param output instead of just debug.

Update Archive::Info->archiveIdList() to return a valid error code instead of unknown.

Add cvtBoolToConstZ() to simplify conversion of boolean to string.

Add cvtZToUInt() to convert string to unsigned int.

Enable -Wstrict-prototypes, -Wpointer-arith, -Wduplicated-branches, -Wvla, and -Wduplicated-cond and update code to conform where necessary.

Rename error-handling variables in Main.pm to conform to standard.

Remove redundant lines from embedded Perl by combining blank lines.

Define cipher magic size with sizeof() rather than using a constant.

Add cvtCharToZ() and macro for debugging char params.

Add strReplaceChr() to String object.

Correct OptionInvalidError to OptionInvalidValueError in boolean option parsing.

Move cryptographic hash functions to C using OpenSSL.

Split log levels into separate header file. Many modules that use debug.h do not need to do logging so this reduces dependencies for those modules.

Auto-generate Makefile with dependencies.

Rename cipher module to the more general crypto.

Update Debian package to add debug symbols to executable.

Convert the not very portable uint type to unsigned int.

Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.

Build libc using links rather than referencing the C files in src directly. The C library builds with different options which should not be reused for the C binary or vice versa.

Make backup.history sync more efficient. Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.

Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.

Improved lock implementation written in C. Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.

Storage object improvements. Convert all functions to variadic functions. Enforce read-only storage. Add storageLocalWrite() helper function. Add storageCopy(), storageExists(), storageMove(), storageNewRead()/storageNewWrite(), storagePathCreate(), storagePathRemove(), storagePathSync(), and storageRemove(). Add StorageFileRead and StorageFileWrite objects. Abstract Posix driver code into a separate module. Call storagePathRemove() from the Perl Posix driver.

Improve String and StringList objects. Add strUpper(), strLower(), strLstExists(), strLstExistsZ(), strChr(), strSub(), strSubN(), and strTrunc().

Improve Buffer object. Add bufNewC(), bufEq() and bufCat(). Only reallocate buffer when the size has changed.

Add pgControlInfo() to read pg_control and determine the version.

Add walSegmentNext() and walSegmentRange().

Error handling improvments. Add THROWP_* macro variants for error handling. These macros allow an ErrorType pointer to be passed and are required for functions that may return different errors based on a parameter. Add _FMT variants for all THROW macros so format types are checked by the compiler.

Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing.

Allow MemContext objects to be copied to a new parent. This makes it easier to create objects and then copy them to another context when they are complete without having to worry about freeing them on error. Update List, StringList, and Buffer to allow moves. Update Ini and Storage to take advantage of moves.

Full branch coverage in C code.

Refactor usec to msec in common/time.c. The implementation provides usec resolution but this is not needed in practice and it makes the interface more complicated due to the extra zeros.

Replace THROW_ON_SYS_ERROR() with THROW_SYS_ERROR(). The former macro was hiding missing branch coverage for critical error handling.

Start work on C handle io object and use it to output help.

Don't copy CFGDEF_NAME_ALT or CFGDEF_INHERIT when processing config option inheritance.

Split debug and assert code into separate headers. Assert can be used earlier because it only depends on the error-handler and not logging. Add ASSERT() macro which is preserved in production builds.

Cleanup C types. Remove typec.h. Order all typdefs above local includes.

Fix header exclusion defines that do not match the general pattern.

Improve Perl configuration. Set config before Main::main() call to avoid secrets being exposed in a stack trace. Move logic for setting defaults to C.

Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.

Port most of Config::Config::configLoad() from Perl to C.

Fix incorrect enum types in config.c that throw warnings under clang.

Enable -Wswitch-enum, -Wconversion, -Wformat=2, -Wformat-nonliteral, and -Wformat-signedness and silence new warnings.

Improve code documentation in config module.

Improve debugging. Add ASSERT_DEBUG() macro for debugging and replace all current assert() calls except in tests that can't use the debug code. Replace remaining NDEBUG blocks with the more granular DEBUG_UNIT. Remove some debug memset() calls in MemContext since valgrind is more useful for these checks.

Add cfgOptionTest() and update cfgOption() calls that are better implemented as cfgOptionTest().

Build with -DNDEBUG by default but disable for testing.

Check int size in common/type.h. This ensures that integers are at least 32-bits without having to run the test suite.

Improve conversion of C exceptions to Exception objects. Colons in the message would prevent all of the message from being loaded into the Exception object.

Implement help command in C.

Implement version command in C.

Config parsing implemented in C and passed to Perl as JSON.

Add Buffer, Ini, KeyValue, List, RegExp, Storage, String, StringList, Variant, VariantList, and Wait objects.

Add command, exit, log, and time modules.

Remove deprecated archive-max-mb option.

Improve MemContext module. Add temporary context blocks and refactor allocation arrays to include allocation size.

Improve error module. Add functions to convert error codes to C errors and handle system errors.

Create a master list of errors in build/error.yaml. The C and Perl errors lists are created automatically by Build.pm so they stay up to date.

Move lock release later in exitSafe() to reduce the chance of a new process starting and acquiring a lock before the old process has exited.

Add 30 second wait loop to lockAcquire() when fail on no lock enabled. This should help prevent processes that are shutting down from interfering with processes that are starting up.

Replace cfgCommandTotal()/cfgOptionTotal() functions with constants. The constants are applicable in more cases and allow the compiler to optimize certain loops more efficiently.

Cleanup usage of internal options. Apply internal to options that need to be read to determine locality but should not appear in the help.

Refactor code to make valgrind happy.

Fix non-compliant formatting for function declarations.

Make backup.history sync more efficient. Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.

Add log-level-stderr option for stanza-* commands.

Minor changes to Manifest module, mostly for test reproducibility.

Fix non-compliant formatting for function declarations.

Add memGrowRaw() to memory context module.

Refactor protocol param generation into a new function. This allows the code to be tested more precisely and doesn't require executing a remote process.

Add list type for options. The hash type was being used for lists with an additional flag (`value-hash`) to indicate that it was not really a hash.

Remove configurable option hints. db-path was the only option with a hint so the feature seemed wasteful. All missing stanza options now output the same hint without needing configuration.

Convert configuration definitions from auto-generated functions to auto-generated data structures.

Add eof to S3 file driver (required for encryption support).

Enable additional warnings for C builds.

Simplify try..catch..finally names. Also wrap in a do...while loop to make sure that no random else is attached to the main if block.

Improve base64 implementation. Different encoded strings could be generated based on compiler optimizations. Even though decoding was still successful the encoded strings did not match the standard.

Disable -Wclobber compiler warning because it is mostly useless but keep the rest of of -Wextra.

Improve protocol error handling. In particular, stop errors are no longer reported as unexpected.

Allow functions with sensitive options to be logged at debug level with redactions. Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.

Replace dynamically built class hierarchies in I/O layer with fixed parent() calls.

Improve labeling for errors in helper processes.

Update C naming conventions.

Use int datatype wherever possible.

Better separation of C source from Perl interface.

Add LibC.template.pm to simplify LibC module generation.

Add C error handler.

Perl error handler recognizes errors thrown from the C library.

Page checksum module uses new C error handler.

Add C memory contexts.

Add base64 encode/decode.

Dynamically generate list of files for C library build.

Break up LibC.xs into separate module files.

Configuration definitions are now pulled from the C library when present.

Add bIgnoreMissing parameter to Local->manifest().

Harden protocol handshake to handle race conditions.

Fixed misleading error message when a file was opened for write in a missing directory.

Change log level of hardlink logging to detail.

Cast size in S3 manifest to integer.

Rename Archive modules to remove redundancy.

Improve S3 error reporting.

Minor optimizations to package loads and ordering for archive-get and archive-push commands.

Refactor storage layer to allow for new repository filesystems using drivers.

Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters.

Move modules in Protocol directory in subdirectories.

Move backup modules into Backup directory.

Refactor Ini.pm to facilitate testing.

The backup and restore commands no longer copy via temp files. In both cases the files are checksummed on resume so there's no danger of partial copies.

Allow functions to accept optional parameters as a hash.

Refactor File->list() and fileList() to accept optional parameters.

Refactor backupLabel() and add unit tests.

Silence some perl critic warnings.

Improved the code and tests for fileManifest() to prevent a possible race condition when files are removed by the database while the manifest is being built.

Refactor FileCommon::fileManifest() and FileCommon::fileStat to be more modular to allow complete branch/statement level coverage testing.

Refactor File and BackupCommon modules to improve test coverage.

Moved File->manifest() into the FileCommon.pm module.

Moved the Archive modules to the Archive directory and split the archive-get and archive-push commands into separate modules.

Split the check command out of the Archive.pm module.

Allow logging to be suppressed via logDisable() and logEnable().

Allow for locks to be taken more than once in the same process without error.

Lock directories can be created when more than one directory level is required.

Clean up optionValid()/optionTest() logic in Lock.pm.

Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.

Represent .gz extension with a constant.

Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.

Refactor process IO and process master/minion code out from the common protocol code.

Fixed alignment issues with multiline logging.

Abstracted code to determine which database cluster is the primary and which are standbys.

Improved consistency and flexibility of the protocol layer by using JSON for all messages.

File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process.

Improved IO->bufferRead to always return requested number of bytes until EOF.

Simplified the result hash of File->manifest(), Db->tablespaceMapGet(), and Db->databaseMapGet().

Improved errors returned from child processes by removing redundant error level and code.

Code cleanup in preparation for improved stanza-create command.

Improved parameter/result logging in debug/trace functions.

Refactor of protocol minions in preparation for the new local minion.

Remove obsolete thread index variable from File() module.

Changed temporary file names to consistently use the .pgbackrest.tmp extension even if the destination file is compressed or has an appended checksum.

Improve ASSERT error handling, safely check eval blocks, and convert $@ to $EVAL_ERROR.

Simplify protocol creation and identifying which host is local/remote.

Removed all OP_* function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__.

Improvements in Db module: separated out connect() function, allow executeSql() calls that do not return data, and improve error handling.

Enhancements to the protocol layer for improved reliability and error handling.

Exceptions are now passed back from threads as messages when possible rather than raised directly.

Temp files created during backup are now placed in the same directory as the target file.

Output lock file name when a lock cannot be acquired to aid in debugging.

Reduce calls to protocolGet() in backup/restore.

The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.

Changed version variable to a constant. It had originally been designed to play nice with a specific packaging tool but that tool was never used.

Log directory create and file open now using FileCommon functions which produce more detailed error messages on failure.

Removed dependency on IO::String module.

Code cleanup and refactoring to standardize on patterns that have evolved over time.

Major refactoring of the protocol layer to support future development.

Removed pg_backrest_remote and added the functionality to pg_backrest as the remote command.

Moved version number out of the VERSION file to Version.pm to better support packaging.

Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple operating systems.

Improved threading model by starting threads early and terminating them late.

Removed dependency on Moose. It wasn't being used extensively and makes for longer startup times.

Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.

Removed dependency on Storable and replaced with a custom ini file implementation.

Numerous other changes that can only be identified with a diff.

Worked on improving error handling in the File object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).