1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

357 Commits

Author SHA1 Message Date
Cynthia Shang
eb30d88b6a Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta.
Contributed by Cynthia Shang.
2018-08-24 16:50:33 -04:00
David Steele
0ed37ab9e7 Update Archive::Info->archiveIdList() to return a valid error code instead of unknown. 2018-08-24 12:13:10 -04:00
David Steele
2276567027 Add log-subprocess option to allow file logging for local and remote subprocesses. 2018-08-22 20:05:49 -04:00
David Steele
8a8738308c Enable -Wvla. 2018-08-22 14:48:37 -04:00
David Steele
6643afe9a8 Add gzip compression/decompression filters for C. 2018-08-14 14:56:59 -04:00
David Steele
3434240097 Remove incompletely implemented online option from the check command.
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.

Reported by Jason O'Donnell.
2018-08-12 19:24:21 -04:00
David Steele
31167d8f98 Enable -Wduplicated-branches, and -Wduplicated-cond. 2018-08-05 08:35:18 -04:00
David Steele
14f21a8f6c Enable -Wpointer-arith. 2018-08-04 19:37:28 -04:00
David Steele
429a356e33 Enable -Wstrict-prototypes and update all void functions to conform. 2018-08-03 19:19:14 -04:00
Cynthia Shang
bec4c176dc Exclude temporary and unlogged relation (table/index) files from backup.
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.

Contributed by Cynthia Shang.
2018-07-30 18:53:34 -04:00
David Steele
666537cbde Update code count for new file types and exclusions. 2018-07-28 08:13:30 -04:00
David Steele
1862630629 Use pre-built images from Docker Hub when the container definition has not changed.
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
2018-07-21 17:02:42 -04:00
David Steele
8568622a6f Add --log-level-test option.
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
2018-07-20 19:03:46 -04:00
David Steele
5dc8a2ec08 storageFileRead() accepts a buffer for output rather than creating one.
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.

Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
2018-07-17 19:01:54 -04:00
Cynthia Shang
0acf705416 Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
Contributed by Cynthia Shang.
2018-07-16 17:25:15 -04:00
David Steele
22adb395db Add zero-length file to mock/all test. 2018-07-05 15:40:50 -04:00
David Steele
db17973cd0 Fix critical bug in resume that resulted in inconsistent backups.
A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".

Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 14:01:57 -04:00
David Steele
3d07e01fe9 Make ls ordering deterministic in mock/all test. 2018-07-01 20:02:37 -04:00
Andrew Schwartz
1bd98b61df Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.
AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.

Fixed by Andrew Schwartz.
2018-07-01 08:17:27 -04:00
David Steele
7b0e65d488 Improve the HTTP client to set content-length to 0 when not specified by the server.
S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.

Suggested by Adam K. Sumner.
2018-06-26 17:27:22 -04:00
David Steele
d55e609959 Use a prebuilt s3 server container for documentation and tests. 2018-06-12 13:43:15 -04:00
David Steele
350b30fa49 Move cryptographic hash functions to C using OpenSSL. 2018-06-11 14:52:26 -04:00
David Steele
835396751f Document generator improvements.
* Build containers from scratch for more accurate testing.
* Allow environment load to be skipped.
* Allow bash wrapping to be skipped.
* Allow forcing a command to run as a user without sudo.
2018-06-10 14:13:56 -04:00
David Steele
a385cb520b Update primary test environment (Vagrant and Docker) to Ubuntu 18.04. 2018-06-06 15:52:28 -04:00
David Steele
d309a85b51 PostgreSQL 11 Beta 1 support. 2018-06-05 08:59:17 -04:00
David Steele
e57840d739 Document generator improvements.
* Allow parameters to be passed when a container is created.
* Allow /etc/hosts update to be skipped (for containers without bash).
2018-05-25 13:42:09 -04:00
David Steele
40093f160c Improve efficiency of C library builds now that they are used only for testing. 2018-05-24 14:01:24 -04:00
David Steele
8eb9921c7c Auto-generate Makefile with dependencies. 2018-05-23 14:57:08 -04:00
David Steele
f7123c1bbf v2.03: Single Executable to Deploy
Bug Fixes:

* Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
* Fix archive write lock being taken for the synchronous archive-get command. (Reported by Uspen.)

Improvements:

* Embed exported C functions and Perl modules directly into the pgBackRest executable.
* Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
* Print total runtime in milliseconds at command end.
2018-05-22 19:53:25 -04:00
David Steele
4c7408ea52 Embed exported C functions and Perl modules directly into the pgBackRest executable. 2018-05-22 12:53:08 -04:00
David Steele
ada5c5c0d9 Fix archive write lock being taken for the synchronous archive-get command.
Reported by Uspen.
2018-05-21 10:46:45 -04:00
David Steele
52bc073234 Add stack trace macros to all functions.
Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
David Steele
abb9651f4c Test harness improvements.
* Allow more than one test to provide coverage for the same module.
* Add option to disable valgrind.
* Add option to disabled coverage.
* Add option to disable debug build.
* Add option to disable compiler optimization.
* Add --dev-test mode.
2018-05-18 06:45:14 -04:00
David Steele
91be372e6a Set log-timestamp=n for integration tests.
This means less filtering of logs needs to be done and new timestamps can be added without adding new filters.
2018-05-11 11:24:38 -04:00
David Steele
a1be8e16cd Update SSL error message test on CentOS 7. 2018-05-11 10:39:36 -04:00
David Steele
790f7c7312 Enforce full branch coverage in C code. 2018-05-05 11:50:11 -04:00
David Steele
c3a8fbe706 Add _FMT variants for all THROW macros so format types are checked by the compiler. 2018-05-03 11:24:29 -04:00
Yogesh Sharma
6a40c916d4 Add repo-s3-token option to allow temporary credentials tokens to be configured.
pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire.

Contributed by Yogesh Sharma.
2018-05-02 14:06:40 -04:00
David Steele
5e090ba305 Fix failure in manifest build when two or more files in PGDATA are linked to the same directory.
Reported by Vitaliy Kukharik.
2018-05-02 12:19:54 -04:00
David Steele
8dbbf2de92 Fix archive-copy throwing "path not found" error for incr/diff backups.
Reported by yummyliu, Vitaliy Kukharik.
2018-05-01 12:37:06 -04:00
David Steele
71ba08f579 Use path list in the backup manifest to do restore path syncs.
Remove recursive path sync functionality since it is no longer used.
2018-05-01 11:05:37 -04:00
David Steele
54dd6f3ed4 Add asynchronous, parallel archive-get.
This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
David Steele
89d3476e32 Refactor archive common functions in preparation for parallel async archive-get. 2018-04-29 10:16:59 -04:00
David Steele
88a633da17 Divide tests into three types (unit, integration, performance).
Many options that were set per test can instead be inferred from the types, i.e. container, c, expect, and individual.

Also finish renaming Perl unit tests with the -perl suffix.
2018-04-24 09:12:25 -04:00
Cynthia Shang
27aa5f756e Add support for additional pgBackRest configuration files.
Configuration files are loaded from the directory specified by the --config-include-path option.

Add --config-path option for overriding the default base path of the --config and --config-include-path option.

Contributed by Cynthia Shang.
2018-04-19 11:09:39 -04:00
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
cf0ec44ede Rename Perl tests so they don't conflict with their C counterparts. 2018-04-17 12:03:03 -04:00
David Steele
dce11ce3cf Generate code counts for all source files.
The source files are also classified by type and purpose.
2018-04-16 15:52:17 -04:00
David Steele
3e22977484 Use pip 9.03 in test VMs.
pip 10 drops support for Python 2.6 which is still used by the older test VMs.
2018-04-16 15:25:50 -04:00
David Steele
2a4ff2f904 Split cfgLoad() into multiple functions to make testing easier.
Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00