mirror of
https://github.com/pgbackrest/pgbackrest.git
synced 2024-12-14 10:13:05 +02:00
c5892d1291
When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part. Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response. Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously). There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory). A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality. Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
761 lines
26 KiB
YAML
761 lines
26 KiB
YAML
# **********************************************************************************************************************************
|
|
# Test Definition
|
|
#
|
|
# Contains definitions for all unit and integration tests.
|
|
#
|
|
# The definitions consist of modules and tests. Modules group together tests that are related and allow common settings for all the
|
|
# tests. Every module and test must have a name. Tests must also have a total. If this total does not match the actual number of
|
|
# runs in a test then an error will be thrown.
|
|
#
|
|
# Most options can be set for modules and tests (test option will override module option if both are set):
|
|
# * db - determines if the test will be run against multiple db versions
|
|
# * coverage - determines coverage for the test
|
|
# full - the module/test provides full coverage for the code module
|
|
# partial - the module/test provides partial coverage for the code module
|
|
# noCode - the code module should not contain any coverable code. If it does an error will be thrown.
|
|
# * define - defines for C code (will also be applied to the test harness)
|
|
# * define-test - defines for the test harness
|
|
# * debugUnitSuppress - don't define DEBUG_UNIT for unit tests -- this is used to test unit test debugging macros
|
|
# * binReq - is the pgbackrest binary required for this test?
|
|
# * containerReq - is this test required to run in a container?
|
|
#
|
|
# Some options are unique to tests:
|
|
# * total - total runs in the test
|
|
# * vm - VMs that the test will be run on
|
|
# * include - modules to include directly into test.c (all files in coverage are automatically included)
|
|
# This is useful when a module's internal data needs to be manipulated for testing but no coverage is added by the test.
|
|
# **********************************************************************************************************************************
|
|
|
|
# **********************************************************************************************************************************
|
|
# Unit tests
|
|
# **********************************************************************************************************************************
|
|
unit:
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: common
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: error
|
|
total: 8
|
|
define-test: -DNO_ERROR -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
|
|
|
|
coverage:
|
|
common/error: full
|
|
common/error.auto: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: assert-on
|
|
total: 1
|
|
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
|
|
|
|
coverage:
|
|
common/assert: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: assert-off
|
|
total: 1
|
|
define: -DNDEBUG
|
|
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
|
|
debugUnitSuppress: true
|
|
|
|
coverage:
|
|
common/assert: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: stack-trace
|
|
total: 4
|
|
define-test: -DNO_LOG -DNO_STACK_TRACE -DNO_MEM_CONTEXT
|
|
|
|
coverage:
|
|
common/stackTrace: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: mem-context
|
|
total: 7
|
|
define-test: -DNO_MEM_CONTEXT -DNO_LOG
|
|
|
|
coverage:
|
|
common/memContext: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: time
|
|
total: 3
|
|
define-test: -DNO_ERROR -DNO_LOG
|
|
|
|
coverage:
|
|
common/time: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: fork
|
|
total: 1
|
|
define-test: -DNO_LOG
|
|
|
|
coverage:
|
|
common/fork: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: log
|
|
total: 5
|
|
define-test: -DNO_LOG
|
|
|
|
coverage:
|
|
common/log: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: debug-off
|
|
total: 2
|
|
define: -DNDEBUG
|
|
define-test: -DNO_LOG
|
|
debugUnitSuppress: true
|
|
|
|
coverage:
|
|
common/debug: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: debug-on
|
|
total: 4
|
|
|
|
coverage:
|
|
common/debug: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: lock
|
|
total: 2
|
|
|
|
coverage:
|
|
common/lock: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: exit
|
|
total: 3
|
|
|
|
coverage:
|
|
common/exit: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: wait
|
|
total: 1
|
|
|
|
coverage:
|
|
common/wait: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-convert
|
|
total: 10
|
|
|
|
coverage:
|
|
common/type/convert: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-object
|
|
total: 1
|
|
|
|
coverage:
|
|
common/type/object: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-list
|
|
total: 4
|
|
|
|
coverage:
|
|
common/type/list: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-string
|
|
total: 26
|
|
|
|
coverage:
|
|
common/type/string: full
|
|
common/type/stringList: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-buffer
|
|
total: 6
|
|
|
|
coverage:
|
|
common/type/buffer: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-variant
|
|
total: 13
|
|
|
|
coverage:
|
|
common/type/variant: full
|
|
common/type/variantList: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-mcv
|
|
total: 1
|
|
|
|
coverage:
|
|
common/type/mcv: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-json
|
|
total: 11
|
|
|
|
coverage:
|
|
common/type/json: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-key-value
|
|
total: 2
|
|
|
|
coverage:
|
|
common/type/keyValue: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type-xml
|
|
total: 1
|
|
|
|
coverage:
|
|
common/type/xml: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: user
|
|
total: 1
|
|
|
|
coverage:
|
|
common/user: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: io
|
|
total: 4
|
|
|
|
coverage:
|
|
common/io/bufferRead: full
|
|
common/io/bufferWrite: full
|
|
common/io/filter/buffer: full
|
|
common/io/filter/filter: full
|
|
common/io/filter/group: full
|
|
common/io/filter/sink: full
|
|
common/io/filter/size: full
|
|
common/io/handleRead: full
|
|
common/io/handleWrite: full
|
|
common/io/io: full
|
|
common/io/read: full
|
|
common/io/write: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: io-tls
|
|
total: 5
|
|
|
|
coverage:
|
|
common/io/tls/client: full
|
|
common/io/tls/session: full
|
|
common/io/socket/client: full
|
|
common/io/socket/common: full
|
|
common/io/socket/session: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: io-http
|
|
total: 5
|
|
|
|
coverage:
|
|
common/io/http/client: full
|
|
common/io/http/common: full
|
|
common/io/http/header: full
|
|
common/io/http/query: full
|
|
common/io/http/request: full
|
|
common/io/http/response: full
|
|
common/io/http/session: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: compress
|
|
total: 5
|
|
|
|
coverage:
|
|
common/compress/bz2/common: full
|
|
common/compress/bz2/compress: full
|
|
common/compress/bz2/decompress: full
|
|
common/compress/gz/common: full
|
|
common/compress/gz/compress: full
|
|
common/compress/gz/decompress: full
|
|
common/compress/lz4/common: full
|
|
common/compress/lz4/compress: full
|
|
common/compress/lz4/decompress: full
|
|
common/compress/zst/common: full
|
|
common/compress/zst/compress: full
|
|
common/compress/zst/decompress: full
|
|
common/compress/helper: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: crypto
|
|
total: 3
|
|
|
|
coverage:
|
|
common/crypto/cipherBlock: full
|
|
common/crypto/common: full
|
|
common/crypto/hash: full
|
|
common/crypto/md5.vendor: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: exec
|
|
total: 1
|
|
|
|
coverage:
|
|
common/exec: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: encode
|
|
total: 1
|
|
|
|
coverage:
|
|
common/encode: full
|
|
common/encode/base64: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: reg-exp
|
|
total: 3
|
|
|
|
coverage:
|
|
common/regExp: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: ini
|
|
total: 4
|
|
|
|
coverage:
|
|
common/ini: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: postgres
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: client
|
|
total: 1
|
|
|
|
coverage:
|
|
postgres/client: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: interface
|
|
total: 9
|
|
|
|
coverage:
|
|
postgres/interface: full
|
|
postgres/interface/page: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: config
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: define
|
|
total: 2
|
|
|
|
coverage:
|
|
config/define: full
|
|
config/define.auto: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: config
|
|
total: 4
|
|
|
|
coverage:
|
|
config/config: full
|
|
config/config.auto: noCode
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: parse
|
|
total: 4
|
|
|
|
coverage:
|
|
config/parse: full
|
|
config/parse.auto: noCode
|
|
|
|
include:
|
|
- common/log
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: load
|
|
total: 4
|
|
|
|
coverage:
|
|
config/load: full
|
|
|
|
include:
|
|
- common/log
|
|
- common/io/socket/common
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: exec
|
|
total: 1
|
|
|
|
coverage:
|
|
config/exec: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: protocol
|
|
total: 1
|
|
|
|
coverage:
|
|
config/protocol: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: storage
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: cifs
|
|
total: 1
|
|
|
|
coverage:
|
|
storage/cifs/storage: full
|
|
storage/posix/storage: full
|
|
storage/helper: full
|
|
|
|
include:
|
|
- storage/storage
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: posix
|
|
total: 21
|
|
|
|
coverage:
|
|
storage/posix/read: full
|
|
storage/posix/storage: full
|
|
storage/posix/write: full
|
|
|
|
# Provide as much coverage as possible for these modules but some coverage needs to be provided by other driver tests
|
|
storage/helper: full
|
|
storage/read: full
|
|
storage/storage: full
|
|
storage/write: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: remote
|
|
total: 9
|
|
containerReq: true
|
|
binReq: true
|
|
|
|
coverage:
|
|
storage/remote/read: full
|
|
storage/remote/protocol: full
|
|
storage/remote/storage: full
|
|
storage/remote/write: full
|
|
storage/helper: full
|
|
storage/storage: full
|
|
|
|
include:
|
|
- storage/read
|
|
- storage/write
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: s3
|
|
total: 3
|
|
|
|
coverage:
|
|
storage/s3/read: full
|
|
storage/s3/storage: full
|
|
storage/s3/write: full
|
|
storage/helper: full
|
|
storage/storage: full
|
|
|
|
include:
|
|
- storage/write
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: protocol
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: protocol
|
|
total: 9
|
|
containerReq: true
|
|
binReq: true
|
|
|
|
coverage:
|
|
protocol/client: full
|
|
protocol/command: full
|
|
protocol/helper: full
|
|
protocol/parallel: full
|
|
protocol/parallelJob: full
|
|
protocol/server: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: info
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: info
|
|
total: 3
|
|
|
|
coverage:
|
|
info/info: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: info-pg
|
|
total: 2
|
|
|
|
coverage:
|
|
info/infoPg: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: info-archive
|
|
total: 2
|
|
|
|
coverage:
|
|
info/infoArchive: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: info-backup
|
|
total: 3
|
|
|
|
coverage:
|
|
info/infoBackup: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: manifest
|
|
total: 6
|
|
|
|
coverage:
|
|
info/manifest: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: db
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: db
|
|
total: 3
|
|
containerReq: true
|
|
binReq: true
|
|
|
|
coverage:
|
|
db/db: full
|
|
db/helper: full
|
|
db/protocol: full
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: command
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive-common
|
|
total: 8
|
|
|
|
coverage:
|
|
command/archive/common: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive-get
|
|
total: 5
|
|
binReq: true
|
|
|
|
coverage:
|
|
command/archive/common: full
|
|
command/archive/get/file: full
|
|
command/archive/get/get: full
|
|
command/archive/get/protocol: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive-push
|
|
total: 4
|
|
binReq: true
|
|
|
|
coverage:
|
|
command/archive/common: full
|
|
command/archive/push/file: full
|
|
command/archive/push/protocol: full
|
|
command/archive/push/push: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
# --test=backup and --test=backup-common must must be run together to get full coverage of backup/common
|
|
- name: backup-common
|
|
total: 3
|
|
|
|
coverage:
|
|
command/backup/common: full
|
|
command/backup/pageChecksum: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: backup
|
|
total: 10
|
|
binReq: true
|
|
|
|
coverage:
|
|
command/backup/backup: full
|
|
command/backup/common: full
|
|
command/backup/file: full
|
|
command/backup/protocol: full
|
|
|
|
include:
|
|
- info/info
|
|
- info/manifest
|
|
- storage/storage
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: check
|
|
total: 4
|
|
containerReq: true
|
|
|
|
coverage:
|
|
command/check/common: full
|
|
command/check/check: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: command
|
|
total: 1
|
|
|
|
coverage:
|
|
command/command: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: control
|
|
total: 3
|
|
|
|
coverage:
|
|
command/control/common: full
|
|
command/control/start: full
|
|
command/control/stop: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: expire
|
|
total: 9
|
|
|
|
coverage:
|
|
command/expire/expire: full
|
|
|
|
include:
|
|
- info/infoBackup
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: help
|
|
total: 4
|
|
|
|
coverage:
|
|
command/help/help: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: info
|
|
total: 3
|
|
|
|
coverage:
|
|
command/info/info: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: local
|
|
total: 1
|
|
|
|
coverage:
|
|
command/local/local: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: remote
|
|
total: 1
|
|
|
|
coverage:
|
|
command/remote/remote: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: restore
|
|
total: 12
|
|
binReq: true
|
|
|
|
coverage:
|
|
command/restore/file: full
|
|
command/restore/protocol: full
|
|
command/restore/restore: full
|
|
|
|
include:
|
|
- common/user
|
|
- info/infoBackup
|
|
- info/manifest
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: stanza
|
|
total: 5
|
|
|
|
coverage:
|
|
command/stanza/common: full
|
|
command/stanza/create: full
|
|
command/stanza/upgrade: full
|
|
command/stanza/delete: full
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: repo
|
|
total: 4
|
|
|
|
coverage:
|
|
command/repo/get: full
|
|
command/repo/ls: full
|
|
command/repo/put: full
|
|
command/repo/rm: full
|
|
|
|
include:
|
|
# command/repo/create is currently for testing purposes only so coverage is not provided except in integration. In the
|
|
# future this will probably be rolled into a custom object store server implementation.
|
|
- command/repo/create
|
|
|
|
# **********************************************************************************************************************************
|
|
# Integration tests
|
|
#
|
|
# Integration tests are not run in a container. They are expected to create their own containers since most integration runs will
|
|
# create more than one. For this reason each run is executed individually.
|
|
# **********************************************************************************************************************************
|
|
integration:
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: mock
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: stanza
|
|
total: 2
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: expire
|
|
total: 2
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive
|
|
total: 2
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive-stop
|
|
total: 2
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: all
|
|
total: 2
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: real
|
|
db: true
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: all
|
|
total: 1
|
|
|
|
# **********************************************************************************************************************************
|
|
# Performance tests
|
|
#
|
|
# Performance tests run in a single container but are more like integration tests than unit tests since they call the pgbackrest
|
|
# executable directly.
|
|
# **********************************************************************************************************************************
|
|
performance:
|
|
|
|
# ********************************************************************************************************************************
|
|
- name: performance
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: archive-perl
|
|
total: 1
|
|
|
|
test:
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: type
|
|
total: 3
|
|
|
|
# ----------------------------------------------------------------------------------------------------------------------------
|
|
- name: storage
|
|
total: 2
|
|
|
|
include:
|
|
- storage/helper
|