db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-13 01:00:23 +02:00
Code Issues Releases Activity
Files
251dbede8ff8a7206986106b8df8b08257cd7ebd
pgbackrest/test/src/module/command/archivePushTest.c

94 lines
4.9 KiB
C
Raw Normal View History

The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Archive Push Command
***********************************************************************************************************************************/
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
#include "storage/driver/posix/storage.h"
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
#include "common/harnessConfig.h"
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
Fix non-compliant formatting for function declarations. Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Create default storage object for testing
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
Storage *storageTest = storageDriverPosixInterface(
storageDriverPosixNew(strNew(testPath()), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, true, NULL));
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// *****************************************************************************************************************************
if (testBegin("cmdArchivePush()"))
{
StringList *argList = strLstNew();
Add separate archive-push-async command. This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation. A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-03-14 13:38:55 +04:00
strLstAddZ(argList, "pgbackrest-bogus");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
strLstAddZ(argList, "--archive-timeout=1");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "archive-push");
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
TEST_ERROR(cmdArchivePush(), ParamRequiredError, "WAL segment to push required");
// -------------------------------------------------------------------------------------------------------------------------
strLstAddZ(argList, "000000010000000100000001");
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Add separate archive-push-async command. This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation. A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-03-14 13:38:55 +04:00
TEST_ERROR(cmdArchivePush(), OptionRequiredError , "===PERL-EMBED-ERROR===");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
// Make sure the process times out when there is nothing to archive
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
storagePathCreateNP(storageTest, strNewFmt("%s/db/archive_status", testPath()));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
strLstAdd(argList, strNewFmt("--spool-path=%s", testPath()));
strLstAddZ(argList, "--archive-async");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
strLstAdd(argList, strNewFmt("--log-path=%s", testPath()));
strLstAdd(argList, strNewFmt("--log-level-file=debug"));
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
strLstAdd(argList, strNewFmt("--pg1-path=%s/db", testPath()));
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
"unable to push WAL segment '000000010000000100000001' asynchronously after 1 second(s)");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Wait for the lock to release
Construct Wait object in milliseconds instead of fractional seconds. The Wait object accepted a double in the constructor for wait time but used TimeMSec internally. This was done for compatibility with the Perl code. Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code. Note that Perl still uses a double for its Wait object so translation is needed in some places. There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
lockAcquire(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), cfgLockType(), 30000, true);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
lockRelease(true);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Call Perl with built-in interpreter instead of execvp(). Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
// Write out a bogus .error file to make sure it is ignored on the first loop
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Remove the archive status path so async will error and not overwrite the bogus error file
storagePathRemoveNP(storageTest, strNewFmt("%s/db/archive_status", testPath()));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
String *errorFile = storagePathNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.error"));
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
storagePutNP(storageNewWriteNP(storageSpoolWrite(), errorFile), bufNewZ("25\n" BOGUS_STR));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Call Perl with built-in interpreter instead of execvp(). Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
TEST_ERROR(cmdArchivePush(), AssertError, BOGUS_STR);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
storageRemoveP(storageTest, errorFile, .errorOnMissing = true);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Call Perl with built-in interpreter instead of execvp(). Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
// Write out a valid ok file and test for success
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Storage object improvements. * Convert all functions to variadic functions. * Enforce read-only storage. * Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat(). * Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
storagePutNP(
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok")), bufNew(0));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
TEST_RESULT_VOID(cmdArchivePush(), "successful push");
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
harnessLogResult("P00 INFO: pushed WAL segment 000000010000000100000001 asynchronously");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageRemoveP(
storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok"), .errorOnMissing = true);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Make sure the process times out when there is nothing to archive and it can't get a lock. This test MUST go last since
// the lock is lost and cannot be closed by the main process.
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(
Construct Wait object in milliseconds instead of fractional seconds. The Wait object accepted a double in the constructor for wait time but used TimeMSec internally. This was done for compatibility with the Perl code. Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code. Note that Perl still uses a double for its Wait object so translation is needed in some places. There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
lockAcquire(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), cfgLockType(), 30000, true), "acquire lock");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
TEST_RESULT_VOID(lockClear(true), "clear lock");
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
"unable to push WAL segment '000000010000000100000001' asynchronously after 1 second(s)");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_RESULT_VOID();
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
}
Reference in New Issue Copy Permalink