db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-04 09:43:08 +02:00
Code Issues Releases Activity

Migrate coverage testing to C and remove dependency on lcov.

Browse Source 
lcov does not seem to be very well maintained and is often not compatible with the version of gcc it ships with until a few months after a new distro is released. In any case, lcov is that not useful for us because it generates reports on all coverage while we are mainly interested in missing coverage during development.

Instead use the JSON output generated by gcov to generate our minimal coverage report and metrics for the documentation.

There are some slight differences in the metrics. The difference in the common module was due to a bug in the old code -- build/common was being added into common as well as being reported separately. The source of the two additional branches in the backup module is unknown but almost certainly down to how exclusions are processed with regular expressions. Since there is additional coverage rather than coverage missing this seems fine.

Since this was pretty much a rewrite it was also a good time to migrate to C.
This commit is contained in:
David Steele 2024-05-31 14:52:07 +10:00
parent 49e252f492
commit 6f562fba60
26 changed files with 1875 additions and 1163 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
CONTRIBUTING.md
View File

@ -30,7 +30,7 @@ This example is based on Ubuntu 20.04, but it should work on many versions of De
pgbackrest-dev => Install development tools
```
sudo apt-get install rsync git devscripts build-essential valgrind lcov autoconf \
sudo apt-get install rsync git devscripts build-essential valgrind autoconf \
autoconf-archive libssl-dev zlib1g-dev libxml2-dev libpq-dev pkg-config \
libxml-checker-perl libyaml-perl libdbd-pg-perl liblz4-dev liblz4-tool \
zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache python3-distutils meson
@ -532,7 +532,7 @@ pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u20
```
> **NOTE:** Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.
Because a test run has not been specified, a coverage report will be generated and written to the local file system under the pgBackRest directory `test/result/coverage/lcov/index.html` and a file with only the highlighted code that has not been covered will be written to `test/result/coverage/coverage.html`.
A coverage report will be generated and written to the local file system under the pgBackRest repository in `test/result/coverage.html`.
If 100 percent code coverage has not been achieved, an error message will be displayed, for example: `ERROR: [125]: c module command/check/check is not fully covered`

2
doc/src/meson.build
View File

@ -19,6 +19,7 @@ subdir('command/help')
# test target
####################################################################################################################################
src_doc = [
'../../src/build/common/json.c',
'../../src/build/common/render.c',
'../../src/build/common/string.c',
'../../src/build/common/xml.c',
@ -38,7 +39,6 @@ src_doc = [
'../../src/common/io/fdWrite.c',
'../../src/common/lock.c',
'../../src/common/stat.c',
'../../src/common/type/json.c',
'../../src/config/config.c',
'../../src/config/parse.c',
'command/build/build.c',

14
doc/xml/auto/metric-coverage-report.auto.xml
View File

@ -71,7 +71,7 @@
<table-row>
<table-cell>command/backup</table-cell>
<table-cell>50/50 (100.0%)</table-cell>
<table-cell>790/790 (100.0%)</table-cell>
<table-cell>792/792 (100.0%)</table-cell>
<table-cell>1896/1896 (100.0%)</table-cell>
</table-row>
@ -161,9 +161,9 @@
<table-row>
<table-cell>common</table-cell>
<table-cell>149/149 (100.0%)</table-cell>
<table-cell>634/634 (100.0%)</table-cell>
<table-cell>1853/1853 (100.0%)</table-cell>
<table-cell>147/147 (100.0%)</table-cell>
<table-cell>630/630 (100.0%)</table-cell>
<table-cell>1829/1829 (100.0%)</table-cell>
</table-row>
<table-row>
@ -357,7 +357,7 @@
<table-row>
<table-cell>TOTAL</table-cell>
<table-cell>1651/1651 (100.0%)</table-cell>
<table-cell>10409/10410 (99.99%)</table-cell>
<table-cell>31143/31143 (100.0%)</table-cell>
<table-cell>1649/1649 (100.0%)</table-cell>
<table-cell>10407/10408 (99.99%)</table-cell>
<table-cell>31119/31119 (100.0%)</table-cell>
</table-row>

4
doc/xml/contributing.xml
View File

@ -91,7 +91,7 @@
<execute user="root" pre="y">
<exe-cmd>
apt-get install rsync git devscripts build-essential valgrind lcov autoconf
apt-get install rsync git devscripts build-essential valgrind autoconf
autoconf-archive libssl-dev zlib1g-dev libxml2-dev libpq-dev pkg-config
libxml-checker-perl libyaml-perl libdbd-pg-perl liblz4-dev liblz4-tool
zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache python3-distutils meson
@ -597,7 +597,7 @@ pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u20
<admonition type="note">Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.</admonition>
<p>Because a test run has not been specified, a coverage report will be generated and written to the local file system under the <backrest/> directory <file>test/result/coverage/lcov/index.html</file> and a file with only the highlighted code that has not been covered will be written to <file>test/result/coverage/coverage.html</file>.</p>
<p>A coverage report will be generated and written to the local file system under the <backrest/> repository in <file>test/result/coverage.html</file>.</p>
<p>If 100 percent code coverage has not been achieved, an error message will be displayed, for example: <code>ERROR: [125]: c module command/check/check is not fully covered</code></p>

21
src/build/common/json.c Normal file
View File

@ -0,0 +1,21 @@
/***********************************************************************************************************************************
JSON Extensions
***********************************************************************************************************************************/
// Include core module
#include "common/type/json.c"
#include "build/common/json.h"
/**********************************************************************************************************************************/
bool
jsonReadUntil(JsonRead *const this, const JsonType type)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM(JSON_READ, this);
FUNCTION_TEST_PARAM(ENUM, type);
FUNCTION_TEST_END();
ASSERT(this != NULL);
FUNCTION_TEST_RETURN(BOOL, jsonReadTypeNextIgnoreComma(this) != type);
}

15
src/build/common/json.h Normal file
View File

@ -0,0 +1,15 @@
/***********************************************************************************************************************************
JSON Extensions
***********************************************************************************************************************************/
#ifndef BUILD_COMMON_JSON_H
#define BUILD_COMMON_JSON_H
#include "common/type/json.h"
/***********************************************************************************************************************************
Functions
***********************************************************************************************************************************/
// Return true unless specified type found
bool jsonReadUntil(JsonRead *this, JsonType type);
#endif

2
test/Dockerfile
View File

@ -12,7 +12,7 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
sudo vim htop jq rsync sysstat curl \
libdbd-pg-perl libxml-checker-perl libyaml-perl \
devscripts build-essential lintian git cloc txt2man debhelper libssl-dev zlib1g-dev libperl-dev libxml2-dev liblz4-dev \
liblz4-tool libpq-dev lcov autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config libyaml-dev libc6-dbg wget meson \
liblz4-tool libpq-dev autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config libyaml-dev libc6-dbg wget meson \
ccache valgrind tzdata uncrustify libssh2-1-dev
# Install Docker

4
test/Vagrantfile vendored
View File

@ -75,7 +75,7 @@ Vagrant.configure(2) do |config|
#-----------------------------------------------------------------------------------------------------------------------
echo 'Install Build Tools' && date
apt-get install -y devscripts build-essential lintian git cloc txt2man debhelper libssl-dev zlib1g-dev libperl-dev \
libxml2-dev liblz4-dev liblz4-tool libpq-dev lcov autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config \
libxml2-dev liblz4-dev liblz4-tool libpq-dev autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config \
libyaml-dev libc6-dbg valgrind meson ccache uncrustify libssh2-1-dev
#-----------------------------------------------------------------------------------------------------------------------
@ -144,7 +144,7 @@ Vagrant.configure(2) do |config|
# Basic environment to build/test pgBackRest using homebrew installed in the local user account.
#-------------------------------------------------------------------------------------------------------------------------------
# git clone --depth=1 https://github.com/Homebrew/brew ~/homebrew
# ~/homebrew/bin/brew install -q pkg-config openssl@1.1 libpq libxml2 libyaml cpanm lcov meson
# ~/homebrew/bin/brew install -q pkg-config openssl@1.1 libpq libxml2 libyaml cpanm meson
# ~/homebrew/bin/cpanm --force --local-lib=~/homebrew/perl5 install YAML::XS XML::Checker::Parser
#
# export PATH="${HOME?}/homebrew/bin:$PATH"

6
test/ci.pl
View File

@ -185,12 +185,6 @@ eval
"gcc ccache python3-distutils git rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev libyaml-dev pkg-config uncrustify" .
" libssh2-1-dev valgrind";
# Add lcov when testing coverage
if (vmCoverageC($strVm))
{
$strPackage .= " lcov";
}
# Extra packages required when testing without containers
if ($strVm eq VM_NONE)
{

24
test/code-count/file-type.yaml
View File

@ -679,6 +679,14 @@ src/build/common/exec.h:
class: build
type: c/h
src/build/common/json.c:
class: build
type: c
src/build/common/json.h:
class: build
type: c/h
src/build/common/regExp.c:
class: build
type: c
@ -2507,10 +2515,6 @@ test/lib/pgBackRestTest/Common/ContainerTest.pm:
class: test/harness
type: perl
test/lib/pgBackRestTest/Common/CoverageTest.pm:
class: test/harness
type: perl
test/lib/pgBackRestTest/Common/DbVersion.pm:
class: test/harness
type: perl
@ -2599,6 +2603,14 @@ test/src/command/test/build.h:
class: test/harness
type: c/h
test/src/command/test/coverage.c:
class: test/harness
type: c
test/src/command/test/coverage.h:
class: test/harness
type: c/h
test/src/command/test/define.c:
class: test/harness
type: c
@ -3207,6 +3219,10 @@ test/src/module/storage/sftpTest.c:
class: test/module
type: c
test/src/module/test/coverageTest.c:
class: test/module
type: c
test/src/module/test/testTest.c:
class: test/module
type: c

21
test/container.yaml
View File

@ -12,19 +12,10 @@
# - docker login -u pgbackrest
# - VM=XXX;DATE=YYYYMMDDX;BASE=pgbackrest/test:${VM?}-base;docker tag ${BASE?} ${BASE?}-${DATE?} && docker push ${BASE?}-${DATE?}
# **********************************************************************************************************************************
20240524A:
20240530A:
x86_64:
u22: eab8001bbbe7c610453ce06adb31ebd971c61592
20240518A:
x86_64:
f40: 5173d773cfff925d4d41bd34029e55775be23c51
20240425A:
x86_64:
d10: fb03907abefd68fe16557b759e7e110e99eda748
u20: c3fc7cc1956c5eb10995119deed7a21b92dd07a7
20240423A:
x86_64:
rh7: 3ba01dc5bbc96eed48287b8e4f52054d4d7030a5
d10: 890b52c1c95ba3dc2d6a6535b8d9e8b273db275d
f40: c52b4f20349d8037d74a4a4543d2e33cd52bb3fa
u20: 9ac8fc879f2a7934b31d9e9df56a6a617be6cbed
u22: 3dc2ae79283af572ebe9d79c8c6a1086f9537233
rh7: bf93465740557a07914137ddc8a2156efde2646b

10
test/define.yaml
View File

@ -254,7 +254,8 @@ unit:
total: 2
coverage:
- common/type/json
- build/common/json
- common/type/json: included
# ----------------------------------------------------------------------------------------------------------------------------
- name: type-key-value
@ -715,6 +716,13 @@ unit:
- test/command/test/define
- test/command/test/test
# ----------------------------------------------------------------------------------------------------------------------------
- name: coverage
total: 1
coverage:
- test/command/test/coverage
# ********************************************************************************************************************************
- name: info

1
test/lib/pgBackRestTest/Common/CodeCountTest.pm
View File

@ -66,7 +66,6 @@ sub codeCountScan
$strFile =~ '^test/result/' ||
$strFile =~ '^test/scratch' ||
$strFile =~ '^test/src/valgrind\.suppress\.' ||
$strFile eq 'test/src/lcov.conf' ||
$strFile eq 'test/uncrustify.cfg');
# Classify the source file

20
test/lib/pgBackRestTest/Common/ContainerTest.pm
View File

@ -439,12 +439,6 @@ sub containerBuild
}
}
# If no specific version of lcov is requested then install the default package
if (!defined($oVm->{$strOS}{&VMDEF_LCOV_VERSION}))
{
$strScript .= ' lcov';
}
#---------------------------------------------------------------------------------------------------------------------------
$strScript .= sectionHeader() .
"# Regenerate SSH keys\n" .
@ -476,20 +470,6 @@ sub containerBuild
" rm -rf /root/${strValgrind}";
}
#---------------------------------------------------------------------------------------------------------------------------
if (defined($oVm->{$strOS}{&VMDEF_LCOV_VERSION}))
{
my $strLCovVersion = $oVm->{$strOS}{&VMDEF_LCOV_VERSION};
my $strLCovPath = "/root/lcov-${strLCovVersion}";
$strScript .= sectionHeader() .
"# Build lcov ${strLCovVersion}\n" .
" wget -q -O - https://github.com/linux-test-project/lcov/releases/download/v${strLCovVersion}/" .
"lcov-${strLCovVersion}.tar.gz | tar zx -C /root && \\\n" .
" make -C ${strLCovPath} install && \\\n" .
" rm -rf ${strLCovPath}";
}
#---------------------------------------------------------------------------------------------------------------------------
if (!$bDeprecated)
{

1070
test/lib/pgBackRestTest/Common/CoverageTest.pm
View File

File diff suppressed because it is too large Load Diff

15
test/lib/pgBackRestTest/Common/JobTest.pm
View File

@ -25,7 +25,6 @@ use pgBackRestDoc::ProjectInfo;
use pgBackRestTest::Common::BuildTest;
use pgBackRestTest::Common::ContainerTest;
use pgBackRestTest::Common::CoverageTest;
use pgBackRestTest::Common::DbVersion;
use pgBackRestTest::Common::DefineTest;
use pgBackRestTest::Common::ExecuteTest;
@ -342,13 +341,13 @@ sub end
# If C code generate coverage info
if ($iExitStatus == 0 && $self->{oTest}->{&TEST_C} && vmCoverageC($self->{oTest}->{&TEST_VM}) && $self->{bCoverageUnit})
{
coverageExtract(
$self->{oStorageTest}, $self->{oTest}->{&TEST_MODULE}, $self->{oTest}->{&TEST_NAME},
$self->{oTest}->{&TEST_VM} ne VM_NONE, $self->{bCoverageSummary},
$self->{oTest}->{&TEST_VM} eq VM_NONE ? undef : $strImage, $self->{strTestPath}, "$self->{strTestPath}/temp",
-e "$self->{strUnitPath}/build/test-unit.p" ?
"$self->{strUnitPath}/build/test-unit.p" : "$self->{strUnitPath}/build/test-unit\@exe",
$self->{strBackRestBase} . '/test/result');
executeTest(
($self->{oTest}->{&TEST_VM} ne VM_NONE ? 'docker exec -i -u ' . TEST_USER . " ${strImage} " : '') .
"gcov --json-format --stdout --branch-probabilities " .
(-e "$self->{strUnitPath}/build/test-unit.p" ?
"$self->{strUnitPath}/build/test-unit.p" : "$self->{strUnitPath}/build/test-unit\@exe") .
'/test.c.gcda > ' . $self->{strBackRestBase} . '/test/result/coverage/raw/' .
$self->{oTest}->{&TEST_MODULE} . '-' . $self->{oTest}->{&TEST_NAME} . '.json');
}
# Record elapsed time

2
test/lib/pgBackRestTest/Common/VmTest.pm
View File

@ -42,8 +42,6 @@ use constant VM_OS_BASE => 'os-base'
push @EXPORT, qw(VM_OS_BASE);
use constant VMDEF_PGSQL_BIN => 'psql-bin';
push @EXPORT, qw(VMDEF_PGSQL_BIN);
use constant VMDEF_LCOV_VERSION => 'lcov-version';
push @EXPORT, qw(VMDEF_LCOV_VERSION);
use constant VMDEF_WITH_LZ4 => 'with-lz4';
push @EXPORT, qw(VMDEF_WITH_LZ4);
use constant VMDEF_WITH_ZST => 'with-zst';

8
test/src/build/config/config.yaml
View File

@ -59,6 +59,12 @@ option:
command:
test: {}
coverage-summary:
type: boolean
default: false
command:
test: {}
neutral-umask:
type: boolean
internal: true
@ -126,8 +132,8 @@ option:
vm-id:
type: integer
required: false
internal: true
default: 0
allow-range: [0, 1024]
command:
test: {}

10
test/src/build/help/help.xml
View File

@ -54,6 +54,16 @@
<example>n</example>
</option>
<option id="coverage-summary">
<summary>Generate code coverage summary.</summary>
<text>
<p>Generate coverage summary used in metrics documentation.</p>
</text>
<example>n</example>
</option>
<option id="log-level">
<summary>Level for harness logging.</summary>

1161
test/src/command/test/coverage.c Normal file
View File

File diff suppressed because it is too large Load Diff

24
test/src/command/test/coverage.h Normal file
View File

@ -0,0 +1,24 @@
/***********************************************************************************************************************************
Coverage Testing and Reporting
***********************************************************************************************************************************/
#ifndef TEST_COMMAND_TEST_COVERAGE_H
#define TEST_COMMAND_TEST_COVERAGE_H
#include "common/type/stringList.h"
/***********************************************************************************************************************************
Functions
***********************************************************************************************************************************/
// Generate coverage summary/report
int testCvgGenerate(
const String *repoPath, const String *pathTest, const String *vm, bool coverageSummary, const StringList *moduleList);
/***********************************************************************************************************************************
Macros for function logging
***********************************************************************************************************************************/
#define FUNCTION_LOG_TEST_COVERAGE_TYPE \
TestCoverage *
#define FUNCTION_LOG_TEST_COVERAGE_FORMAT(value, buffer, bufferSize) \
objNameToLog(value, "TestCoverage", buffer, bufferSize)
#endif

12
test/src/main.c
View File

@ -8,6 +8,7 @@ Main
#include "command/command.h"
#include "command/exit.h"
#include "command/help/help.h"
#include "command/test/coverage.h"
#include "command/test/test.h"
#include "common/debug.h"
#include "common/log.h"
@ -68,6 +69,9 @@ main(int argListSize, const char *argList[])
// Test
// -----------------------------------------------------------------------------------------------------------------
case cfgCmdTest:
{
// Run a single test
if (cfgOptionTest(cfgOptVmId))
{
cmdTest(
cfgOptionStr(cfgOptRepoPath), cfgOptionStr(cfgOptTestPath), cfgOptionStr(cfgOptVm),
@ -76,6 +80,14 @@ main(int argListSize, const char *argList[])
logLevelEnum(cfgOptionStrId(cfgOptLogLevelTest)), cfgOptionBool(cfgOptLogTimestamp),
cfgOptionStrNull(cfgOptTz), cfgOptionBool(cfgOptCoverage), cfgOptionBool(cfgOptProfile),
cfgOptionBool(cfgOptOptimize), cfgOptionBool(cfgOptBackTrace));
}
// Top-level test
else
{
result = testCvgGenerate(
cfgOptionStr(cfgOptRepoPath), cfgOptionStr(cfgOptTestPath), cfgOptionStr(cfgOptVm),
cfgOptionBool(cfgOptCoverageSummary), cfgCommandParam()) > 0;
}
break;
}

3
test/src/meson.build
View File

@ -20,6 +20,7 @@ subdir('command/help')
####################################################################################################################################
src_test = [
'../../src/build/common/exec.c',
'../../src/build/common/json.c',
'../../src/build/common/render.c',
'../../src/build/common/string.c',
'../../src/build/common/yaml.c',
@ -35,10 +36,10 @@ src_test = [
'../../src/common/io/fdWrite.c',
'../../src/common/lock.c',
'../../src/common/stat.c',
'../../src/common/type/json.c',
'../../src/config/config.c',
'../../src/config/parse.c',
'command/test/build.c',
'command/test/coverage.c',
'command/test/define.c',
'command/test/test.c',
'config/load.c',

2
test/src/module/common/typeJsonTest.c
View File

@ -185,6 +185,7 @@ testRun(void)
TEST_RESULT_BOOL(jsonReadKeyExpectStrId(read, STRID5("key5", 0xee4ab0)), false, "do not expect key5");
TEST_ERROR(jsonReadKeyRequireStrId(read, STRID5("key5", 0xee4ab0)), JsonFormatError, "required key 'key5' not found");
TEST_ERROR(jsonReadKeyRequireZ(read, "key5"), JsonFormatError, "required key 'key5' not found");
TEST_RESULT_BOOL(jsonReadUntil(read, jsonTypeObjectEnd), false, "read until object end");
TEST_RESULT_VOID(jsonReadObjectEnd(read), "object end");
TEST_RESULT_STR_Z(jsonReadStr(read), "abc/", "str");
@ -193,6 +194,7 @@ testRun(void)
TEST_RESULT_STRLST_Z(jsonReadStrLst(read), "a\nb\n", "str list");
TEST_RESULT_UINT(jsonReadUInt(read), 123, "uint");
TEST_RESULT_UINT(jsonReadUInt64(read), 18446744073709551615U, "uint64");
TEST_RESULT_BOOL(jsonReadUntil(read, jsonTypeArrayEnd), true, "read until array end");
TEST_RESULT_INT(jsonReadInt(read), -1, "int");
TEST_RESULT_INT(jsonReadInt64(read), -9223372036854775807L, "int64");
TEST_RESULT_VOID(jsonReadSkip(read), "skip");

524
test/src/module/test/coverageTest.c Normal file
View File

@ -0,0 +1,524 @@
/***********************************************************************************************************************************
Test Coverage Testing and Reporting
***********************************************************************************************************************************/
#include "storage/posix/storage.h"
#include "common/harnessStorage.h"
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
static void
testRun(void)
{
FUNCTION_HARNESS_VOID();
const Storage *const storageTest = storagePosixNewP(STRDEF(TEST_PATH), .write = true);
// *****************************************************************************************************************************
if (testBegin("testCvgGenerate()"))
{
const String *const pathRepo = STRDEF(TEST_PATH "/repo");
const String *const pathTest = STRDEF(TEST_PATH "/test");
HRN_STORAGE_PUT_Z(
storageTest, "repo/test/define.yaml",
"unit:\n"
" - name: common\n"
" test:\n"
" - name: none\n"
" total: 1\n"
"\n"
" - name: error\n"
" total: 1\n"
" coverage:\n"
" - common/error/sub/error.vendor: included\n"
" - doc/common/error/error: included\n"
" - test/common/error/error\n"
" - test/noCode: noCode\n"
"\n"
" - name: log\n"
" total: 1\n"
" coverage:\n"
" - common/log\n"
" - common/log2\n"
" - test/common/error/error\n"
"\n"
"integration:\n"
" - name: mock\n"
" test:\n"
" - name: all\n"
" total: 2\n"
"\n"
"performance:\n"
" - name: performance\n"
" test:\n"
" - name: type\n"
" total: 1\n");
HRN_STORAGE_PUT_Z(
storageTest, "test/repo/test/src/common/error/error.c",
/* 01 */ "FN_EXTERN const ErrorType *\n"
/* 02 */ "errorCodeIsError(const int code)\n"
/* 03 */ "{\n"
/* 04 */ " if (code == 0)\n"
/* 05 */ " return false;\n"
/* 06 */ "\n"
/* 07 */ " return true;\n"
/* 08 */ "}\n");
HRN_STORAGE_PUT_Z(
storageTest, "test/repo/doc/src/common/error/error.c",
/* 01 */ "int\n"
/* 02 */ "returnCode(int code)\n"
/* 03 */ "{\n"
/* 04 */ " return code;\n"
/* 05 */ "}\n");
HRN_STORAGE_PUT_Z(
storageTest, "test/repo/src/common/error/sub/error.vendor.c.inc",
/* 01 */ "int\n"
/* 02 */ "returnCode(int code)\n"
/* 03 */ "{\n"
/* 04 */ " return code;\n"
/* 05 */ "}\n");
HRN_STORAGE_PUT_Z(
storageTest, "repo/test/result/coverage/raw/common-type-error.json",
"{"
// {uncrustify_off - indentation}
"\"gcc_version\": \"11.4.0\","
"\"files\": ["
"{"
"\"lines\": ["
"{"
"\"branches\": ["
"{"
"\"count\": 1"
"},"
"{"
"\"count\": 0"
"}"
"],"
"\"count\": 1,"
"\"line_number\": 4"
"},"
"{"
"\"count\": 1,"
"\"line_number\": 5"
"},"
"{"
"\"count\": 0,"
"\"line_number\": 7"
"}"
"],"
"\"functions\": ["
"{"
"\"start_line\": 2,"
"\"name\": \"errorCodeIsError\","
"\"execution_count\": 1,"
"\"end_line\": 8"
"}"
"],"
"\"file\": \"../../../repo/test/src/common/error/error.c\""
"},"
"{"
"\"lines\": ["
"{"
"\"count\": 1,"
"\"line_number\": 4"
"}"
"],"
"\"functions\": ["
"{"
"\"start_line\": 2,"
"\"name\": \"returnCode\","
"\"execution_count\": 1,"
"\"end_line\": 5"
"}"
"],"
"\"file\": \"../../../repo/test/src/common/error/sub/error.vendor.c.inc\""
"},"
"{"
"\"lines\": ["
"{"
"\"count\": 1,"
"\"line_number\": 4"
"}"
"],"
"\"functions\": ["
"{"
"\"start_line\": 2,"
"\"name\": \"returnCode\","
"\"execution_count\": 1,"
"\"end_line\": 5"
"}"
"],"
"\"file\": \"../../../repo/doc/src/common/error/error.c\""
"}"
"]"
// {uncrustify_on}
"}");
HRN_STORAGE_PUT_Z(
storageTest, "test/repo/src/common/log.c",
/* 01 */ "static void\n"
/* 02 */ "logAnySet(void)\n"
/* 03 */ "{\n"
/* 04 */ " FUNCTION_TEST_VOID();\n"
/* 05 */ "\n"
/* 06 */ " logLevelAny = logLevelStdOut;\n"
/* 07 */ "\n"
/* 08 */ " if (logLevelStdErr > logLevelAny)\n"
/* 09 */ " logLevelAny = logLevelStdErr;\n"
/* 10 */ "\n"
/* 11 */ " if (logLevelFile > logLevelAny && logFdFile != -1)\n"
/* 12 */ " logLevelAny = logLevelFile;\n"
/* 13 */ "\n"
/* 14 */ " FUNCTION_TEST_RETURN_VOID();\n"
/* 15 */ "}\n"
/* 16 */ "\n"
/* 17 */ "FN_EXTERN bool\n"
/* 18 */ "logAny(const LogLevel logLevel)\n"
/* 19 */ "{\n"
/* 20 */ " FUNCTION_TEST_BEGIN();\n"
/* 21 */ " FUNCTION_TEST_PARAM(ENUM, logLevel);\n"
/* 22 */ " FUNCTION_TEST_END();\n"
/* 23 */ "\n"
/* 24 */ " ASSERT(logLevel);\n"
/* 25 */ "\n"
/* 26 */ " FUNCTION_TEST_RETURN(BOOL, logLevel); // {uncovered - x}\n"
/* 27 */ " x = 1;\n"
/* 28 */ "\n"
/* 29 */ "{\n"
/* 30 */ "}\n"
/* 31 */ "\n"
/* 32 */ "\n"
/* 33 */ "\n"
/* 34 */ "\n"
/* 35 */ "{\n"
/* 36 */ "}\n"
/* 37 */ "\n"
/* 38 */ " x = 2;\n"
/* 39 */ "}\n");
HRN_STORAGE_PUT_Z(
storageTest, "test/repo/src/common/log2.c",
/* 01 */ "int\n"
/* 02 */ "returnCode(int code)\n"
/* 03 */ "{\n"
/* 04 */ " if (code == 0)\n"
/* 05 */ " return code;\n"
/* 06 */ "}\n");
HRN_STORAGE_PUT_Z(
storageTest, "repo/test/result/coverage/raw/common-log.json",
"{"
// {uncrustify_off - indentation}
"\"gcc_version\": \"11.4.0\","
"\"files\": ["
"{"
"\"lines\": ["
"{"
"\"branches\": ["
"{"
"\"count\": 0"
"},"
"{"
"\"count\": 1"
"}"
"],"
"\"count\": 1,"
"\"line_number\": 4"
"},"
"{"
"\"count\": 0,"
"\"line_number\": 5"
"},"
"{"
"\"count\": 1,"
"\"line_number\": 7"
"}"
"],"
"\"functions\": ["
"{"
"\"start_line\": 2,"
"\"name\": \"errorCodeIsError\","
"\"execution_count\": 1,"
"\"end_line\": 8"
"}"
"],"
"\"file\": \"../../../repo/test/src/common/error/error.c\""
"},"
"{"
"\"lines\": ["
"{"
"\"branches\": [],"
"\"count\": 0,"
"\"line_number\": 4"
"},"
"{"
"\"branches\": [],"
"\"count\": 1,"
"\"line_number\": 6"
"},"
"{"
"\"branches\": ["
"{"
"\"fallthrough\": true,"
"\"count\": 1"
"},"
"{"
"\"count\": 0"
"}"
"],"
"\"count\": 7,"
"\"junk\": 0,"
"\"line_number\": 8"
"},"
"{"
"\"branches\": [],"
"\"count\": 0,"
"\"line_number\": 22"
"},"
"{"
"\"branches\": ["
"{"
"\"count\": 1"
"},"
"{"
"\"count\": 0"
"}"
"],"
"\"count\": 1,"
"\"line_number\": 24"
"},"
"{"
"\"branches\": [],"
"\"count\": 0,"
"\"line_number\": 26"
"},"
"{"
"\"branches\": [],"
"\"count\": 0,"
"\"line_number\": 27"
"},"
"{"
"\"branches\": [],"
"\"count\": 0,"
"\"line_number\": 38"
"}"
"],"
"\"functions\": ["
"{"
"\"blocks\": 18,"
"\"start_line\": 2,"
"\"name\": \"logAnySet\","
"\"execution_count\": 1,"
"\"end_line\": 15"
"},"
"{"
"\"blocks\": 18,"
"\"start_line\": 18,"
"\"name\": \"logAny\","
"\"execution_count\": 0,"
"\"end_line\": 33"
"}"
"],"
"\"junk\": 0,"
"\"file\": \"../../../repo/src/common/log.c\""
"},"
"{"
"\"lines\": ["
"{"
"\"branches\": ["
"{"
"\"count\": 0"
"},"
"{"
"\"count\": 1"
"}"
"],"
"\"count\": 1,"
"\"line_number\": 4"
"},"
"{"
"\"count\": 1,"
"\"line_number\": 5"
"}"
"],"
"\"functions\": ["
"{"
"\"start_line\": 2,"
"\"name\": \"returnCode\","
"\"execution_count\": 1,"
"\"end_line\": 6"
"}"
"],"
"\"file\": \"../../../repo/test/src/common/log2.c\""
"}"
"]"
// {uncrustify_on}
"}");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("common/none");
StringList *moduleList = strLstNew();
strLstAddZ(moduleList, "common/none");
TEST_RESULT_INT(testCvgGenerate(pathRepo, pathTest, STRDEF("none"), false, moduleList), 0, "generate");
TEST_STORAGE_GET(
storageTest, "repo/test/result/coverage/coverage.html",
TEST_CVG_HTML_PRE
TEST_CVG_HTML_TOC_PRE
TEST_CVG_HTML_TOC_POST
TEST_CVG_HTML_POST);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("common/error");
moduleList = strLstNew();
strLstAddZ(moduleList, "common/error");
TEST_RESULT_INT(testCvgGenerate(pathRepo, pathTest, STRDEF("vm"), false, moduleList), 0, "generate");
TEST_RESULT_LOG("P00 WARN: module 'test/common/error/error' did not have all tests run required for coverage");
TEST_STORAGE_GET(
storageTest, "repo/test/result/coverage/coverage.html",
TEST_CVG_HTML_PRE
TEST_CVG_HTML_TOC_PRE
TEST_CVG_HTML_TOC_COVERED_PRE "doc/src/common/error/error.c" TEST_CVG_HTML_TOC_COVERED_POST
TEST_CVG_HTML_TOC_COVERED_PRE "src/common/error/sub/error.vendor.c.inc" TEST_CVG_HTML_TOC_COVERED_POST
TEST_CVG_HTML_TOC_POST
TEST_CVG_HTML_POST);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("common/error and common/log summary");
moduleList = strLstNew();
strLstAddZ(moduleList, "common/error");
strLstAddZ(moduleList, "common/log");
TEST_RESULT_INT(testCvgGenerate(pathRepo, pathTest, STRDEF("vm"), true, moduleList), 2, "generate");
TEST_RESULT_LOG(
"P00 WARN: module 'src/common/log.c' is not fully covered (3/8 lines, 1/2 branches)\n"
"P00 WARN: module 'src/common/log2.c' is not fully covered (2/2 lines, 1/2 branches)");
TEST_STORAGE_GET(
storageTest, "repo/doc/xml/auto/metric-coverage-report.auto.xml",
"<table-row>\n"
" <table-cell>common</table-cell>\n"
" <table-cell>2/3 (66.67%)</table-cell>\n"
" <table-cell>2/4 (50.00%)</table-cell>\n"
" <table-cell>5/10 (50.00%)</table-cell>\n"
"</table-row>\n"
"\n"
"<table-row>\n"
" <table-cell>common/error/sub</table-cell>\n"
" <table-cell>1/1 (100.0%)</table-cell>\n"
" <table-cell>---</table-cell>\n"
" <table-cell>1/1 (100.0%)</table-cell>\n"
"</table-row>\n"
"\n"
"<table-row>\n"
" <table-cell>TOTAL</table-cell>\n"
" <table-cell>3/4 (75.00%)</table-cell>\n"
" <table-cell>2/4 (50.00%)</table-cell>\n"
" <table-cell>6/11 (54.55%)</table-cell>\n"
"</table-row>\n");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("common/error and common/log");
TEST_RESULT_INT(testCvgGenerate(pathRepo, pathTest, STRDEF("vm"), false, moduleList), 2, "generate");
TEST_RESULT_LOG(
"P00 WARN: module 'src/common/log.c' is not fully covered (4/8 lines, 1/2 branches)\n"
"P00 WARN: module 'src/common/log2.c' is not fully covered (2/2 lines, 1/2 branches)");
TEST_STORAGE_GET(
storageTest, "repo/test/result/coverage/coverage.html",
TEST_CVG_HTML_PRE
TEST_CVG_HTML_TOC_PRE
TEST_CVG_HTML_TOC_COVERED_PRE "doc/src/common/error/error.c" TEST_CVG_HTML_TOC_COVERED_POST
TEST_CVG_HTML_TOC_COVERED_PRE "src/common/error/sub/error.vendor.c.inc" TEST_CVG_HTML_TOC_COVERED_POST
TEST_CVG_HTML_TOC_UNCOVERED_PRE "src/common/log.c" TEST_CVG_HTML_TOC_UNCOVERED_MID "src/common/log.c"
TEST_CVG_HTML_TOC_UNCOVERED_POST
TEST_CVG_HTML_TOC_UNCOVERED_PRE "src/common/log2.c" TEST_CVG_HTML_TOC_UNCOVERED_MID "src/common/log2.c"
TEST_CVG_HTML_TOC_UNCOVERED_POST
TEST_CVG_HTML_TOC_COVERED_PRE "test/src/common/error/error.c" TEST_CVG_HTML_TOC_COVERED_POST
TEST_CVG_HTML_TOC_POST
TEST_CVG_HTML_RPT_PRE "src/common/log.c" TEST_CVG_HTML_RPT_MID1 "src/common/log.c" TEST_CVG_HTML_RPT_MID2
TEST_CVG_HTML_RPT_LINE_PRE "1" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "static void" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "2" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "logAnySet(void)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "3" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "{" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "4" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE_UNCOVERED " FUNCTION_TEST_VOID();" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "5" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "6" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " logLevelAny = logLevelStdOut;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "7" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "8" TEST_CVG_HTML_RPT_BRANCH_UNCOVERED_PRE "[+ -]" TEST_CVG_HTML_RPT_BRANCH_UNCOVERED_POST
TEST_CVG_HTML_RPT_CODE " if (logLevelStdErr > logLevelAny)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "9" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " logLevelAny = logLevelStdErr;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "10" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "11" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " if (logLevelFile > logLevelAny && logFdFile != -1)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "12" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " logLevelAny = logLevelFile;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_SKIP
TEST_CVG_HTML_RPT_LINE_PRE "17" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "FN_EXTERN bool" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "18" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "logAny(const LogLevel logLevel)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "19" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "{" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "20" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " FUNCTION_TEST_BEGIN();" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "21" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " FUNCTION_TEST_PARAM(ENUM, logLevel);" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "22" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE_UNCOVERED " FUNCTION_TEST_END();" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "23" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "24" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " ASSERT(logLevel);" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "25" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "26" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " FUNCTION_TEST_RETURN(BOOL, logLevel); // {uncovered - x}" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "27" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE_UNCOVERED " x = 1;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_SKIP
TEST_CVG_HTML_RPT_LINE_PRE "38" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE_UNCOVERED " x = 2;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_POST
TEST_CVG_HTML_RPT_PRE "src/common/log2.c" TEST_CVG_HTML_RPT_MID1 "src/common/log2.c" TEST_CVG_HTML_RPT_MID2
TEST_CVG_HTML_RPT_LINE_PRE "1" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "int" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "2" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "returnCode(int code)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "3" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE "{" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "4" TEST_CVG_HTML_RPT_BRANCH_UNCOVERED_PRE "[- +]" TEST_CVG_HTML_RPT_BRANCH_UNCOVERED_POST
TEST_CVG_HTML_RPT_CODE " if (code == 0)" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_LINE_PRE "5" TEST_CVG_HTML_RPT_BRANCH_COVERED
TEST_CVG_HTML_RPT_CODE " return code;" TEST_CVG_HTML_RPT_LINE_POST
TEST_CVG_HTML_RPT_POST
TEST_CVG_HTML_POST);
}
FUNCTION_HARNESS_RETURN_VOID();
}

43
test/test.pl
View File

@ -37,7 +37,6 @@ use pgBackRestDoc::ProjectInfo;
use pgBackRestTest::Common::BuildTest;
use pgBackRestTest::Common::CodeCountTest;
use pgBackRestTest::Common::ContainerTest;
use pgBackRestTest::Common::CoverageTest;
use pgBackRestTest::Common::DefineTest;
use pgBackRestTest::Common::ExecuteTest;
use pgBackRestTest::Common::JobTest;
@ -410,16 +409,12 @@ eval
" ${strTestPath}/data-*");
$oStorageTest->pathCreate("${strTestPath}/temp", {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
# Remove old lcov dirs -- do it this way so the dirs stay open in finder/explorer, etc.
executeTest("rm -rf ${strBackRestBase}/test/result/coverage/lcov/*");
# Overwrite the C coverage report so it will load but not show old coverage
$oStorageTest->pathCreate(
"${strBackRestBase}/test/result/coverage", {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
$oStorageBackRest->put(
"${strBackRestBase}/test/result/coverage/coverage.html",
"<center>[ " . ($bNoCoverage ? "No Coverage Testing" : "Generating Coverage Report") . " ]</center>");
executeTest("rm -rf ${strBackRestBase}/test/result/coverage/lcov");
# Copy C code for coverage tests
if (vmCoverageC($strVm) && !$bDryRun)
@ -1071,24 +1066,50 @@ eval
# Write out coverage info and test coverage
#-------------------------------------------------------------------------------------------------------------------------------
my $iUncoveredCodeModuleTotal = 0;
my $bUncoveredCodeModule = false;
if (vmCoverageC($strVm) && !$bNoCoverage && !$bDryRun && $iTestFail == 0)
{
$iUncoveredCodeModuleTotal = coverageValidateAndGenerate(
$oyTestRun, $oStorageBackRest, !$bNoCoverageReport, $bCoverageSummary, $strTestPath, "${strTestPath}/temp",
"${strBackRestBase}/test/result", "${strBackRestBase}/doc/xml/auto");
my $strModuleList;
foreach my $hTest (@{$oyTestRun})
{
$strModuleList .= ' ' . $hTest->{&TEST_MODULE} . '/' . $hTest->{&TEST_NAME};
}
my $oExec = new pgBackRestTest::Common::ExecuteTest(
"${strBuildPath}/test/src/test-pgbackrest --log-level=warn --vm=${strVm} --repo-path=${strBackRestBase}" .
" --test-path=${strTestPath}" . ($bCoverageSummary ? ' --coverage-summary' : '') . " test${strModuleList}",
{bShowOutputAsync => true, bSuppressError => true});
$oExec->begin();
my $iResult = $oExec->end();
if ($iResult <= 1)
{
$bUncoveredCodeModule = $iResult == 1;
if (!$bUncoveredCodeModule)
{
&log(INFO, "tested modules have full coverage");
}
}
else
{
&log(ERROR, "TEST SUBCOMMAND RETURNED ERROR CODE $iResult");
exit $iResult;
}
}
# Print test info and exit
#-------------------------------------------------------------------------------------------------------------------------------
&log(INFO,
($bDryRun ? 'DRY RUN COMPLETED' : 'TESTS COMPLETED') . ($iTestFail == 0 ? ' SUCCESSFULLY' .
($iUncoveredCodeModuleTotal == 0 ? '' : " WITH ${iUncoveredCodeModuleTotal} MODULE(S) MISSING COVERAGE") :
(!$bUncoveredCodeModule ? '' : " WITH MODULE(S) MISSING COVERAGE") :
" WITH ${iTestFail} FAILURE(S)") . ($iTestRetry == 0 ? '' : ", ${iTestRetry} RETRY(IES)") .
($bNoLogTimestamp ? '' : ' (' . (time() - $lStartTime) . 's)'));
exit 1 if ($iTestFail > 0 || ($iUncoveredCodeModuleTotal > 0 && !$bCoverageSummary));
exit 1 if ($iTestFail > 0 || ($bUncoveredCodeModule && !$bCoverageSummary));
exit 0;
}