db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-07 00:35:37 +02:00
Code Issues Releases Activity
Files
fda105ebd18f050d8071f62add2933ecbff8b8a7
pgbackrest/test/src/module/performance/storageTest.c

380 lines
18 KiB
C
Raw Normal View History

Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
/***********************************************************************************************************************************
Storage Performance
Test the performance of various storage functions, in particular when implemented remotely.
Generally speaking, the starting values should be high enough to "blow up" in terms of execution time if there are performance
problems without taking very long if everything is running smoothly. These starting values can then be scaled up for profiling and
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
stress testing as needed.
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
***********************************************************************************************************************************/
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
Move dummy storage driver to test harness. The dummy driver is the basis for creating test storage drivers so it makes sense to locate it in the harness where all tests can access it.
2020-07-25 08:44:41 -04:00
#include "common/harnessStorage.h"
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
#include "common/crypto/hash.h"
#include "common/compress/gz/compress.h"
#include "common/compress/lz4/compress.h"
#include "common/io/filter/filter.intern.h"
#include "common/io/filter/sink.h"
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
#include "common/io/bufferRead.h"
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
#include "common/io/bufferWrite.h"
Rename handle to fd. Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd. There were also plenty of places fd was used so now all uses are consistent. The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
#include "common/io/fdRead.h"
#include "common/io/fdWrite.h"
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
#include "common/io/io.h"
Move common/object.h to common/type/object.h. This header does not contain a type but is used to define types so this seems like a better location.
2020-03-30 20:52:57 -04:00
#include "common/type/object.h"
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
#include "protocol/client.h"
#include "protocol/server.h"
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
#include "storage/posix/storage.h"
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
#include "storage/remote/protocol.h"
/***********************************************************************************************************************************
Dummy callback functions
***********************************************************************************************************************************/
static void
storageTestDummyInfoListCallback(void *data, const StorageInfo *info)
{
(void)data;
(void)info;
// Do some work in the mem context to blow up the total time if this is not efficient
memResize(memNew(16), 32);
}
/***********************************************************************************************************************************
Driver to test storageInfoList
***********************************************************************************************************************************/
typedef struct
{
STORAGE_COMMON_MEMBER;
uint64_t fileTotal;
} StorageTestPerfInfoList;
static bool
storageTestPerfInfoList(
Simplify storage driver info and list functions. The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent. Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact. Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs. Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
THIS_VOID, const String *path, StorageInfoLevel level, StorageInfoListCallback callback, void *callbackData,
StorageInterfaceInfoListParam param)
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
{
THIS(StorageTestPerfInfoList);
Simplify storage driver info and list functions. The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent. Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact. Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs. Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
(void)path; (void)level; (void)param;
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
MEM_CONTEXT_TEMP_BEGIN()
{
MEM_CONTEXT_TEMP_RESET_BEGIN()
{
for (uint64_t fileIdx = 0; fileIdx < this->fileTotal; fileIdx++)
{
Add file name to make performance/storage test more realistic. Also add timing information.
2020-07-31 16:18:56 -04:00
callback(callbackData, &(StorageInfo){.exists = true, .name = STRDEF("name")});
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
MEM_CONTEXT_TEMP_RESET(1000);
}
}
MEM_CONTEXT_TEMP_END();
}
MEM_CONTEXT_TEMP_END();
return this->fileTotal != 0;
}
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
/***********************************************************************************************************************************
Test filter to simulate throughput via rate limiting
***********************************************************************************************************************************/
typedef struct TestIoRate
{
MemContext *memContext; // Mem context of filter
uint64_t timeBegin; // Time when filter started processing data in ms
uint64_t byteTotal; // Total bytes processed
uint64_t bytesPerSec; // Rate in bytes per second to enforce
} TestIoRate;
static void
testIoRateProcess(THIS_VOID, const Buffer *input)
{
THIS(TestIoRate);
// Determine the elapsed time since the filter began processing data. The begin time is not set in the constructor because an
// unknown amount of time can elapse between the filter being created and acually used.
uint64_t timeElapsed = 0;
if (this->timeBegin == 0)
this->timeBegin = timeMSec();
else
timeElapsed = timeMSec() - this->timeBegin;
// Add buffer used to the byte total
this->byteTotal += bufUsed(input);
// Determine how many ms these bytes should take to go through the filter and sleep if greater than elapsed time
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
uint64_t timeRate = this->byteTotal * MSEC_PER_SEC / this->bytesPerSec;
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
if (timeElapsed < timeRate)
sleepMSec(timeRate - timeElapsed);
}
static IoFilter *
testIoRateNew(uint64_t bytesPerSec)
{
IoFilter *this = NULL;
MEM_CONTEXT_NEW_BEGIN("TestIoRate")
{
TestIoRate *driver = memNew(sizeof(TestIoRate));
*driver = (TestIoRate)
{
.memContext = memContextCurrent(),
.bytesPerSec = bytesPerSec,
};
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
this = ioFilterNewP(strNew("TestIoRate"), driver, NULL, .in = testIoRateProcess);
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
}
MEM_CONTEXT_NEW_END();
return this;
}
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
testRun(void)
{
FUNCTION_HARNESS_VOID();
// *****************************************************************************************************************************
if (testBegin("storageInfoList()"))
{
// One million files represents a fairly large cluster
CHECK(testScale() <= 2000);
uint64_t fileTotal = (uint64_t)1000000 * testScale();
HARNESS_FORK_BEGIN()
{
HARNESS_FORK_CHILD_BEGIN(0, true)
{
// Create a basic configuration so the remote storage driver can determine the storage type
StringList *argList = strLstNew();
strLstAddZ(argList, "--" CFGOPT_STANZA "=test");
strLstAddZ(argList, "--" CFGOPT_PROCESS "=0");
strLstAddZ(argList, "--" CFGOPT_REMOTE_TYPE "=" PROTOCOL_REMOTE_TYPE_REPO);
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleRemote, argList);
// Create a driver to test remote performance of storageInfoList() and inject it into storageRepo()
StorageTestPerfInfoList driver =
{
.interface = storageInterfaceTestDummy,
.fileTotal = fileTotal,
};
driver.interface.infoList = storageTestPerfInfoList;
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
storageHelper.storageRepo = memNew(sizeof(Storage *));
storageHelper.storageRepo[0] = storageNew(
STRDEF("TEST"), STRDEF("/"), 0, 0, false, NULL, &driver, driver.interface);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
// Setup handler for remote storage protocol
Rename handle to fd. Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd. There were also plenty of places fd was used so now all uses are consistent. The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
IoRead *read = ioFdReadNew(strNew("storage server read"), HARNESS_FORK_CHILD_READ(), 60000);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
ioReadOpen(read);
Move file descriptor read/write ready into IoRead/IoWrite. Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea. This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select(). Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
IoWrite *write = ioFdWriteNew(strNew("storage server write"), HARNESS_FORK_CHILD_WRITE(), 1000);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
ioWriteOpen(write);
ProtocolServer *server = protocolServerNew(strNew("storage test server"), strNew("test"), read, write);
protocolServerHandlerAdd(server, storageRemoteProtocol);
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
protocolServerProcess(server, NULL);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
}
HARNESS_FORK_CHILD_END();
HARNESS_FORK_PARENT_BEGIN()
{
// Create client
Rename handle to fd. Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd. There were also plenty of places fd was used so now all uses are consistent. The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
IoRead *read = ioFdReadNew(strNew("storage client read"), HARNESS_FORK_PARENT_READ_PROCESS(0), 60000);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
ioReadOpen(read);
Move file descriptor read/write ready into IoRead/IoWrite. Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea. This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select(). Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
IoWrite *write = ioFdWriteNew(strNew("storage client write"), HARNESS_FORK_PARENT_WRITE_PROCESS(0), 1000);
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
ioWriteOpen(write);
ProtocolClient *client = protocolClientNew(strNew("storage test client"), strNew("test"), read, write);
// Create remote storage
Storage *storageRemote = storageRemoteNew(
STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, false, NULL, client, 1);
Add file name to make performance/storage test more realistic. Also add timing information.
2020-07-31 16:18:56 -04:00
TimeMSec timeBegin = timeMSec();
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
// Storage info list
TEST_RESULT_VOID(
storageInfoListP(storageRemote, NULL, storageTestDummyInfoListCallback, NULL),
"list %" PRIu64 " remote files", fileTotal);
Add file name to make performance/storage test more realistic. Also add timing information.
2020-07-31 16:18:56 -04:00
TEST_LOG_FMT("list transferred in %ums", (unsigned int)(timeMSec() - timeBegin));
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
// Free client
protocolClientFree(client);
}
HARNESS_FORK_PARENT_END();
}
HARNESS_FORK_END();
}
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
// *****************************************************************************************************************************
if (testBegin("benchmark filters"))
{
// 4MB buffers are the current default
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
ioBufferSizeSet(4 * 1024 * 1024);
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
// 1MB is a fairly normal table size
CHECK(testScale() <= 1024 * 1024 * 1024);
uint64_t blockTotal = (uint64_t)1 * testScale();
// Set iteration
unsigned int iteration = 1;
// Set rate
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
uint64_t rateIn = 0; // MB/s (0 disables)
uint64_t rateOut = 0; // MB/s (0 disables)
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
// Get the sample pages from disk
Convert storagePosixNew() to storagePosixNewP(). An upcoming feature requires new parameters for storagePosixNew() and this causes a lot of churn because almost every test creates a Posix storage object. Some refactoring in the tests might reduce this duplication but storagePosixNew() is collecting a lot of parameters so converting to storagePosixNewP() makes sense in any case. There are relatively few call sites in the core code but they still benefit from better readability after this change.
2020-04-30 11:01:38 -04:00
Buffer *block = storageGetP(storageNewReadP(storagePosixNewP(STR(testRepoPath())), STRDEF("test/data/filecopy.table.bin")));
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
ASSERT(bufUsed(block) == 1024 * 1024);
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
// Build the input buffer
Add casts to performance/storage test for 32-bit architectures.
2021-01-24 15:15:50 -05:00
Buffer *input = bufNew((size_t)blockTotal * bufSize(block));
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
for (unsigned int blockIdx = 0; blockIdx < blockTotal; blockIdx++)
memcpy(bufPtr(input) + (blockIdx * bufSize(block)), bufPtr(block), bufSize(block));
bufUsedSet(input, bufSize(input));
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE_FMT(
Add casts to performance/storage test for 32-bit architectures.
2021-01-24 15:15:50 -05:00
"%u iteration(s) of %zuMiB with %" PRIu64 "MB/s input, %" PRIu64 "MB/s output", iteration,
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
bufUsed(input) / bufUsed(block), rateIn, rateOut);
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
#define BENCHMARK_BEGIN() \
IoWrite *write = ioBufferWriteNew(bufNew(0)); \
#define BENCHMARK_FILTER_ADD(filter) \
ioFilterGroupAdd(ioWriteFilterGroup(write), filter);
#define BENCHMARK_END(addTo) \
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
if (rateOut != 0) \
ioFilterGroupAdd(ioWriteFilterGroup(write), testIoRateNew(rateOut * 1000 * 1000)); \
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
ioFilterGroupAdd(ioWriteFilterGroup(write), ioSinkNew()); \
ioWriteOpen(write); \
\
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
IoRead *read = ioBufferReadNew(input); \
if (rateIn != 0) \
ioFilterGroupAdd(ioReadFilterGroup(read), testIoRateNew(rateIn * 1000 * 1000)); \
ioReadOpen(read); \
\
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
uint64_t benchMarkBegin = timeMSec(); \
\
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
Buffer *buffer = bufNew(ioBufferSize()); \
\
do \
{ \
ioRead(read, buffer); \
ioWrite(write, buffer); \
bufUsedZero(buffer); \
} \
while (!ioReadEof(read)); \
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
\
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
ioReadClose(read); \
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
ioWriteClose(write); \
\
addTo += timeMSec() - benchMarkBegin;
// Start totals to 1ms just in case something takes 0ms to run
uint64_t copyTotal = 1;
Add MD5 hash filter to performance tests.
2020-05-18 19:02:11 -04:00
uint64_t md5Total = 1;
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
uint64_t sha1Total = 1;
uint64_t sha256Total = 1;
uint64_t gzip6Total = 1;
uint64_t lz41Total = 1;
for (unsigned int idx = 0; idx < iteration; idx++)
{
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("copy iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_END(copyTotal);
}
MEM_CONTEXT_TEMP_END();
Add MD5 hash filter to performance tests.
2020-05-18 19:02:11 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("md5 iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_FILTER_ADD(cryptoHashNew(HASH_TYPE_MD5_STR));
BENCHMARK_END(md5Total);
}
MEM_CONTEXT_TEMP_END();
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("sha1 iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_FILTER_ADD(cryptoHashNew(HASH_TYPE_SHA1_STR));
BENCHMARK_END(sha1Total);
}
MEM_CONTEXT_TEMP_END();
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("sha256 iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_FILTER_ADD(cryptoHashNew(HASH_TYPE_SHA256_STR));
BENCHMARK_END(sha256Total);
}
MEM_CONTEXT_TEMP_END();
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("gzip -6 iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_FILTER_ADD(gzCompressNew(6));
BENCHMARK_END(gzip6Total);
}
MEM_CONTEXT_TEMP_END();
// -------------------------------------------------------------------------------------------------------------------------
TEST_LOG_FMT("lz4 -1 iteration %u", idx + 1);
MEM_CONTEXT_TEMP_BEGIN()
{
BENCHMARK_BEGIN();
BENCHMARK_FILTER_ADD(lz4CompressNew(1));
BENCHMARK_END(lz41Total);
}
MEM_CONTEXT_TEMP_END();
}
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("results");
Improve storage filter performance tests. Improve the accuracy of the calculations in several areas with better integer expressions. Make the input buffer size configurable. Previously it was always 1mb, i.e. block size. Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
#define TEST_RESULT(name, total) \
TEST_LOG_FMT( \
"%s time %" PRIu64"ms, avg time %" PRIu64"ms, avg throughput: %" PRIu64 "MB/s", name, total, total / iteration, \
iteration * blockTotal * 1024 * 1024 * 1000 / total / 1000000);
TEST_RESULT("copy", copyTotal);
TEST_RESULT("md5", md5Total);
TEST_RESULT("sha1", sha1Total);
TEST_RESULT("sha256", sha256Total);
TEST_RESULT("gzip -6", gzip6Total);
TEST_RESULT("lz4 -1", lz41Total);
Add storage filter performance test. This test allows the important storage filters to be benchmarked by MiB/s.
2020-03-29 21:25:48 -04:00
}
Add performance/storage test. The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote. The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.
2020-03-26 21:05:36 -04:00
FUNCTION_HARNESS_RESULT_VOID();
}
Reference in New Issue Copy Permalink