db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-05-31 22:49:46 +02:00
Code Issues Releases Activity

Storage object improvements.

Browse Source 
* Convert all functions to variadic functions.
* Enforce read-only storage.
* Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat().
* Add StorageFile object and storageOpenRead()/storageOpenWrite().
This commit is contained in:
David Steele 2018-04-03 12:25:21 -04:00
parent 90f980fe91
commit 93fdb98d15
30 changed files with 1141 additions and 287 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
CODING.md
View File

@ -31,7 +31,7 @@ Type names use camel case with the first letter upper case:
**#define Constants**
`#define` constants should be all caps with `_` separators.
```
```c
#DEFINE MY_CONSTANT "STRING"
```
The value should be aligned at column 69 whenever possible.
@ -41,7 +41,7 @@ This type of constant should mostly be used for strings. Use enums whenever poss
**Enum Constants**
Enum elements follow the same case rules as variables. They are strongly typed so this shouldn't present any confusion.
```
```c
typedef enum
{
cipherModeEncrypt,
@ -55,12 +55,12 @@ Note the comma after the last element. This reduces diff churn when new elements
Macro names should be upper-case with underscores between words. Macros (except simple constants) should be avoided whenever possible as they make code less clear and test coverage harder to measure.
Macros should follow the format:
```
```c
#define MACRO(paramName1, paramName2) \
<code>
```
If the macro defines a block it should look like:
```
```c
#define MACRO_2(paramName1, paramName2) \
{ \
<code> \
@ -70,6 +70,8 @@ Continuation characters should be aligned at column 132 (unlike the examples abo
To avoid conflicts, variables in a macro will be named `[macro name]_[var name]`, e.g. `TEST_RESULT_resultExpected`. Variables that need to be accessed in wrapped code should be provided accessor macros.
[Variadic functions](#variadic-functions) are an exception to the capitalization rule.
#### Begin / End
Use `Begin` / `End` for names rather than `Start` / `Finish`, etc.
@ -85,19 +87,19 @@ Use `New` / `Free` for constructors and destructors rather than `Create` / `Dest
C allows braces to be excluded for a single statement. However, braces should be used when the control statement (if, while, etc.) spans more than one line or the statement to be executed spans more than one line.
No braces needed:
```
```c
if (condition)
return value;
```
Braces needed:
```
```c
if (conditionThatUsesEntireLine1 &&
conditionThatUsesEntireLine2)
{
return value;
}
```
```
```c
if (condition)
{
return
@ -122,6 +124,38 @@ Don't use a macro when a function could be used instead. Macros make it hard to
Object-oriented programming is used extensively. The object pointer is always referred to as `this`.
### Variadic Functions
Variadic functions can take a variable number of parameters. While the `printf()` pattern is variadic, it is not very flexible in terms of optional parameters given in any order.
This project implements variadic functions using macros (which are exempt from the normal macro rule of being all caps). A typical variadic function definition:
```c
typedef struct StoragePathCreateParam
{
bool errorOnExists;
bool noParentCreate;
mode_t mode;
} StoragePathCreateParam;
#define storagePathCreateP(this, pathExp, ...) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){__VA_ARGS__})
#define storagePathCreateNP(this, pathExp) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){0})
void storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param);
```
Continuation characters should be aligned at column 132 (unlike the example above that has been shortened for display purposes).
This function can be called without variable parameters:
```c
storagePathCreateNP(storageLocal(), "/tmp/pgbackrest");
```
Or with variable parameters:
```c
storagePathCreateP(storageLocal(), "/tmp/pgbackrest", .errorOnExists = true, .mode = 0777);
```
If the majority of functions in a module or object are variadic it is best to provide macros for all functions even if they do not have variable parameters. Do not use the base function when variadic macros exist.
## Testing
### Uncoverable/Uncovered Code
@ -129,7 +163,7 @@ Object-oriented programming is used extensively. The object pointer is always re
#### Uncoverable Code
The `uncoverable` keyword marks code that can never be covered. For instance, a function that never returns because it always throws a error. Uncoverable code should be rare to non-existent outside the common libraries and test code.
```
```c
} // {uncoverable - function throws error so never returns}
```
Subsequent code that is uncoverable for the same reason is marked with `// {+uncoverable}`.
@ -137,7 +171,7 @@ Subsequent code that is uncoverable for the same reason is marked with `// {+unc
#### Uncovered Code
Marks code that is not tested for one reason or another. This should be kept to a minimum and an excuse given for each instance.
```
```c
exit(EXIT_FAILURE); // {uncovered - test harness does not support non-zero exit}
```
Subsequent code that is uncovered for the same reason is marked with `// {+uncovered}`.

60
doc/xml/coding.xml
View File

@ -45,7 +45,7 @@
<p><code>#define</code> constants should be all caps with <id>_</id> separators.</p>
<code-block>
<code-block type="c">
#DEFINE MY_CONSTANT "STRING"
</code-block>
@ -57,7 +57,7 @@
<p>Enum elements follow the same case rules as variables. They are strongly typed so this shouldn't present any confusion.</p>
<code-block>
<code-block type="c">
typedef enum
{
cipherModeEncrypt,
@ -75,14 +75,14 @@ typedef enum
<p>Macros should follow the format:</p>
<code-block>
<code-block type="c">
#define MACRO(paramName1, paramName2) \
&lt;code&gt;
</code-block>
<p>If the macro defines a block it should look like:</p>
<code-block>
<code-block type="c">
#define MACRO_2(paramName1, paramName2) \
{ \
&lt;code&gt; \
@ -92,6 +92,8 @@ typedef enum
<p>Continuation characters should be aligned at column 132 (unlike the examples above that have been shortened for display purposes).</p>
<p>To avoid conflicts, variables in a macro will be named <id>[macro name]_[var name]</id>, e.g. <id>TEST_RESULT_resultExpected</id>. Variables that need to be accessed in wrapped code should be provided accessor macros.</p>
<p><link section="/language-elements/variadic-functions">Variadic functions</link> are an exception to the capitalization rule.</p>
</section>
<section id="begin-end">
@ -117,14 +119,14 @@ typedef enum
<p>No braces needed:</p>
<code-block>
<code-block type="c">
if (condition)
return value;
</code-block>
<p>Braces needed:</p>
<code-block>
<code-block type="c">
if (conditionThatUsesEntireLine1 &amp;&amp;
conditionThatUsesEntireLine2)
{
@ -132,7 +134,7 @@ if (conditionThatUsesEntireLine1 &amp;&amp;
}
</code-block>
<code-block>
<code-block type="c">
if (condition)
{
return
@ -166,6 +168,46 @@ if (condition)
<p>Object-oriented programming is used extensively. The object pointer is always referred to as <id>this</id>.</p>
</section>
<section id="variadic-functions">
<title>Variadic Functions</title>
<p>Variadic functions can take a variable number of parameters. While the <code>printf()</code> pattern is variadic, it is not very flexible in terms of optional parameters given in any order.</p>
<p>This project implements variadic functions using macros (which are exempt from the normal macro rule of being all caps). A typical variadic function definition:</p>
<code-block type="c">
typedef struct StoragePathCreateParam
{
bool errorOnExists;
bool noParentCreate;
mode_t mode;
} StoragePathCreateParam;
#define storagePathCreateP(this, pathExp, ...) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){__VA_ARGS__})
#define storagePathCreateNP(this, pathExp) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){0})
void storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param);
</code-block>
<p>Continuation characters should be aligned at column 132 (unlike the example above that has been shortened for display purposes).</p>
<p>This function can be called without variable parameters:</p>
<code-block type="c">
storagePathCreateNP(storageLocal(), "/tmp/pgbackrest");
</code-block>
<p>Or with variable parameters:</p>
<code-block type="c">
storagePathCreateP(storageLocal(), "/tmp/pgbackrest", .errorOnExists = true, .mode = 0777);
</code-block>
<p>If the majority of functions in a module or object are variadic it is best to provide macros for all functions even if they do not have variable parameters. Do not use the base function when variadic macros exist.</p>
</section>
</section>
<section id="testing">
@ -179,7 +221,7 @@ if (condition)
<p>The <id>uncoverable</id> keyword marks code that can never be covered. For instance, a function that never returns because it always throws a error. Uncoverable code should be rare to non-existent outside the common libraries and test code.</p>
<code-block>
<code-block type="c">
} // {uncoverable - function throws error so never returns}
</code-block>
@ -191,7 +233,7 @@ if (condition)
<p>Marks code that is not tested for one reason or another. This should be kept to a minimum and an excuse given for each instance.</p>
<code-block>
<code-block type="c">
exit(EXIT_FAILURE); // {uncovered - test harness does not support non-zero exit}
</code-block>

4
doc/xml/release.xml
View File

@ -15,6 +15,10 @@
<release date="XXXX-XX-XX" version="2.02dev" title="UNDER DEVELOPMENT">
<release-core-list>
<release-development-list>
<release-item>
<p>Storage object improvements. Convert all functions to variadic functions. Enforce read-only storage. Add <code>storageLocalWrite()</code> helper function. Add <code>storageExists()</code>, <code>storagePathCreate()</code>, <code>storageRemove()</code>, and <code>storageStat()</code>. Add <code>StorageFile</code> object and <code>storageOpenRead()</code>/<code>storageOpenWrite()</code>.</p>
</release-item>
<release-item>
<p>Improve branch coverage in C code.</p>
</release-item>

4
libc/Makefile.PL
View File

@ -93,12 +93,14 @@ my @stryCFile =
'common/type/stringList.c',
'common/type/variant.c',
'common/type/variantList.c',
'common/wait.c',
'config/config.c',
'config/define.c',
'config/load.c',
'config/parse.c',
'perl/config.c',
'postgres/pageChecksum.c',
'storage/file.c',
'storage/helper.c',
'storage/storage.c',
);
@ -117,7 +119,7 @@ WriteMakefile
AUTHOR => 'David Steele <david@pgbackrest.org>',
CCFLAGS => join(' ', qw(
-Wfatal-errors -Wall -Wextra -Wwrite-strings -Wno-clobbered
-Wfatal-errors -Wall -Wextra -Wwrite-strings -Wno-clobbered -Wno-missing-field-initializers
-o $@
-std=c99
-D_FILE_OFFSET_BITS=64

4
src/Makefile
View File

@ -17,7 +17,8 @@ COPT = -O2
CINCLUDE = -I.
# Compile warnings
CWARN = -Wfatal-errors -Wall -Wextra -Wwrite-strings -Wno-clobbered -Wswitch-enum -Wconversion -Wformat=2 -Wformat-nonliteral
CWARN = -Wfatal-errors -Wall -Wextra -Wwrite-strings -Wswitch-enum -Wconversion -Wformat=2 -Wformat-nonliteral \
-Wno-clobbered -Wno-missing-field-initializers
# Automatically generate Perl compile options for the local system
CPERL = `perl -MExtUtils::Embed -e ccopts`
@ -78,6 +79,7 @@ SRCS = \
config/parse.c \
perl/config.c \
perl/exec.c \
storage/file.c \
storage/helper.c \
storage/storage.c \
main.c

8
src/command/archive/push/push.c
View File

@ -28,8 +28,8 @@ walStatus(const String *walSegment, bool confessOnError)
MEM_CONTEXT_TEMP_BEGIN()
{
StringList *fileList = storageList(
storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT), strNewFmt("^%s\\.(ok|error)$", strPtr(walSegment)), true);
StringList *fileList = storageListP(
storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT), .expression = strNewFmt("^%s\\.(ok|error)$", strPtr(walSegment)));
if (fileList != NULL && strLstSize(fileList) > 0)
{
@ -38,14 +38,14 @@ walStatus(const String *walSegment, bool confessOnError)
{
THROW(
AssertError, "multiple status files found in '%s' for WAL segment '%s'",
strPtr(storagePath(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT))), strPtr(walSegment));
strPtr(storagePathNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT))), strPtr(walSegment));
}
// Get the status file content
const String *statusFile = strLstGet(fileList, 0);
String *content = strNewBuf(
storageGet(storageSpool(), strNewFmt("%s/%s", STORAGE_SPOOL_ARCHIVE_OUT, strPtr(statusFile)), false));
storageGetNP(storageOpenReadNP(storageSpool(), strNewFmt("%s/%s", STORAGE_SPOOL_ARCHIVE_OUT, strPtr(statusFile)))));
// Get the code and message if the file has content
int code = 0;

2
src/common/ini.c
View File

@ -214,7 +214,7 @@ iniLoad(Ini *this, const String *fileName)
MEM_CONTEXT_TEMP_BEGIN()
{
iniParse(this, strNewBuf(storageGet(storageLocal(), this->fileName, false)));
iniParse(this, strNewBuf(storageGetNP(storageOpenReadNP(storageLocal(), this->fileName))));
}
MEM_CONTEXT_TEMP_END();
}

16
src/common/type/string.c
View File

@ -110,7 +110,7 @@ strNewN(const char *string, size_t size)
}
/***********************************************************************************************************************************
Return the file part of a string (i.e. everthing after the last / or the entire string if there is no /)
Return the file part of a string (i.e. everything after the last / or the entire string if there is no /)
***********************************************************************************************************************************/
String *
strBase(const String *this)
@ -292,6 +292,20 @@ strFirstLower(String *this)
return this;
}
/***********************************************************************************************************************************
Return the path part of a string (i.e. everything before the last / or "" if there is no /)
***********************************************************************************************************************************/
String *
strPath(const String *this)
{
const char *end = this->buffer + this->size;
while (end > this->buffer && *(end - 1) != '/')
end--;
return strNewN(this->buffer, end - this->buffer <= 1 ? (size_t)(end - this->buffer) : (size_t)(end - this->buffer - 1));
}
/***********************************************************************************************************************************
Return string ptr
***********************************************************************************************************************************/

1
src/common/type/string.h
View File

@ -33,6 +33,7 @@ bool strEq(const String *this, const String *compare);
bool strEqZ(const String *this, const char *compare);
String *strFirstUpper(String *this);
String *strFirstLower(String *this);
String *strPath(const String *this);
const char *strPtr(const String *this);
size_t strSize(const String *this);
String *strTrim(String *this);

3
src/common/wait.c
View File

@ -102,5 +102,6 @@ Free the wait
void
waitFree(Wait *this)
{
memContextFree(this->memContext);
if (this != NULL)
memContextFree(this->memContext);
}

3
src/config/parse.c
View File

@ -255,7 +255,8 @@ configParse(unsigned int argListSize, const char *argList[])
configFile = strNew(cfgDefOptionDefault(commandDefId, cfgOptionDefIdFromId(cfgOptConfig)));
// Load the ini file
Buffer *buffer = storageGet(storageLocal(), configFile, !parseOptionList[cfgOptConfig].found);
Buffer *buffer = storageGetNP(
storageOpenReadP(storageLocal(), configFile, .ignoreMissing = !parseOptionList[cfgOptConfig].found));
// Load the config file if it was found
if (buffer != NULL)

83
src/storage/file.c Normal file
View File

@ -0,0 +1,83 @@
/***********************************************************************************************************************************
Storage File
***********************************************************************************************************************************/
#include "common/debug.h"
#include "common/memContext.h"
#include "storage/file.h"
/***********************************************************************************************************************************
Storage file structure
***********************************************************************************************************************************/
struct StorageFile
{
MemContext *memContext;
const Storage *storage;
String *name;
StorageFileType type;
void *data;
};
/***********************************************************************************************************************************
Create a new storage file
This object expects its context to be created in advance. This is so the calling function can add whatever data it wants without
required multiple functions and contexts to make it safe.
***********************************************************************************************************************************/
StorageFile *storageFileNew(const Storage *storage, String *name, StorageFileType type, void *data)
{
ASSERT_DEBUG(storage != NULL);
ASSERT_DEBUG(name != NULL);
ASSERT_DEBUG(data != NULL);
StorageFile *this = memNew(sizeof(StorageFile));
this->memContext = memContextCurrent();
this->storage = storage;
this->name = name;
this->type = type;
this->data = data;
return this;
}
/***********************************************************************************************************************************
Get file data
***********************************************************************************************************************************/
void *
storageFileData(const StorageFile *this)
{
ASSERT_DEBUG(this != NULL);
return this->data;
}
/***********************************************************************************************************************************
Get file name
***********************************************************************************************************************************/
const String *
storageFileName(const StorageFile *this)
{
ASSERT_DEBUG(this != NULL);
return this->name;
}
/***********************************************************************************************************************************
Get file storage object
***********************************************************************************************************************************/
const Storage *
storageFileStorage(const StorageFile *this)
{
ASSERT_DEBUG(this != NULL);
return this->storage;
}
/***********************************************************************************************************************************
Free the file
***********************************************************************************************************************************/
void
storageFileFree(const StorageFile *this)
{
if (this != NULL)
memContextFree(this->memContext);
}

37
src/storage/file.h Normal file
View File

@ -0,0 +1,37 @@
/***********************************************************************************************************************************
Storage File
***********************************************************************************************************************************/
#ifndef STORAGE_FILE_H
#define STORAGE_FILE_H
#include "common/type/string.h"
/***********************************************************************************************************************************
Types of storage files, i.e. read or write. The storage module does not allow files to be opened for both read and write since this
is generally not supported by object stores.
***********************************************************************************************************************************/
typedef enum
{
storageFileTypeRead,
storageFileTypeWrite,
} StorageFileType;
/***********************************************************************************************************************************
Storage file object
***********************************************************************************************************************************/
typedef struct StorageFile StorageFile;
#include "storage/storage.h"
/***********************************************************************************************************************************
Functions
***********************************************************************************************************************************/
StorageFile *storageFileNew(const Storage *storage, String *name, StorageFileType type, void *data);
void *storageFileData(const StorageFile *this);
const String *storageFileName(const StorageFile *this);
const Storage *storageFileStorage(const StorageFile *this);
void storageFileFree(const StorageFile *this);
#endif

35
src/storage/helper.c
View File

@ -1,7 +1,7 @@
/***********************************************************************************************************************************
Storage Helper
***********************************************************************************************************************************/
#include "string.h"
#include <string.h>
#include "common/memContext.h"
#include "config/config.h"
@ -16,6 +16,7 @@ static MemContext *memContextStorageHelper = NULL;
Cache for local storage
***********************************************************************************************************************************/
static Storage *storageLocalData = NULL;
static Storage *storageLocalWriteData = NULL;
/***********************************************************************************************************************************
Cache for spool storage
@ -51,7 +52,7 @@ storageLocal()
{
MEM_CONTEXT_BEGIN(memContextStorageHelper)
{
storageLocalData = storageNew(strNew("/"), STORAGE_PATH_MODE_DEFAULT, 65536, NULL);
storageLocalData = storageNewNP(strNew("/"));
}
MEM_CONTEXT_END();
}
@ -59,6 +60,28 @@ storageLocal()
return storageLocalData;
}
/***********************************************************************************************************************************
Get a writable local storage object
This should be used very sparingly. If writes are not needed then always use storageLocal() or a specific storage object instead.
***********************************************************************************************************************************/
const Storage *
storageLocalWrite()
{
storageHelperInit();
if (storageLocalWriteData == NULL)
{
MEM_CONTEXT_BEGIN(memContextStorageHelper)
{
storageLocalWriteData = storageNewP(strNew("/"), .write = true);
}
MEM_CONTEXT_END();
}
return storageLocalWriteData;
}
/***********************************************************************************************************************************
Get a spool storage object
***********************************************************************************************************************************/
@ -86,14 +109,16 @@ Get a spool storage object
const Storage *
storageSpool()
{
storageHelperInit();
if (storageSpoolData == NULL)
{
MEM_CONTEXT_BEGIN(memContextStorageHelper)
{
storageSpoolStanza = strDup(cfgOptionStr(cfgOptStanza));
storageSpoolData = storageNew(
cfgOptionStr(cfgOptSpoolPath), STORAGE_PATH_MODE_DEFAULT, (size_t)cfgOptionInt(cfgOptBufferSize),
(StoragePathExpressionCallback)storageSpoolPathExpression);
storageSpoolData = storageNewP(
cfgOptionStr(cfgOptSpoolPath), .bufferSize = (size_t)cfgOptionInt(cfgOptBufferSize),
.pathExpressionFunction = storageSpoolPathExpression, .write = true);
}
MEM_CONTEXT_END();
}

4
src/storage/helper.h
View File

@ -14,7 +14,9 @@ Spool storage path constants
/***********************************************************************************************************************************
Functions
***********************************************************************************************************************************/
const Storage *storageSpool();
const Storage *storageLocal();
const Storage *storageLocalWrite();
const Storage *storageSpool();
#endif

419
src/storage/storage.c
View File

@ -6,11 +6,12 @@ Storage Manager
#include <fcntl.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include "common/debug.h"
#include "common/memContext.h"
#include "common/regExp.h"
#include "common/wait.h"
#include "storage/storage.h"
/***********************************************************************************************************************************
@ -18,49 +19,100 @@ Storage structure
***********************************************************************************************************************************/
struct Storage
{
const String *path;
int mode;
MemContext *memContext;
String *path;
mode_t modeFile;
mode_t modePath;
size_t bufferSize;
bool write;
StoragePathExpressionCallback pathExpressionFunction;
};
/***********************************************************************************************************************************
Storage mem context
Storage file data - holds the file handle. This should eventually be moved to the Posix/CIFS driver.
***********************************************************************************************************************************/
static MemContext *storageMemContext = NULL;
typedef struct StorageFileDataPosix
{
MemContext *memContext;
int handle;
} StorageFileDataPosix;
#define STORAGE_DATA(file) \
((StorageFileDataPosix *)storageFileData(file))
/***********************************************************************************************************************************
Debug Asserts
***********************************************************************************************************************************/
// Check that commands that write are not allowed unless the storage is writable
#define ASSERT_STORAGE_ALLOWS_WRITE() \
ASSERT(this->write == true)
/***********************************************************************************************************************************
New storage object
***********************************************************************************************************************************/
Storage *
storageNew(const String *path, int mode, size_t bufferSize, StoragePathExpressionCallback pathExpressionFunction)
storageNew(const String *path, StorageNewParam param)
{
Storage *result = NULL;
Storage *this = NULL;
// Path is required
if (path == NULL)
THROW(AssertError, "storage base path cannot be null");
// If storage mem context has not been initialized yet
if (storageMemContext == NULL)
{
MEM_CONTEXT_BEGIN(memContextTop())
{
storageMemContext = memContextNew("storage");
}
MEM_CONTEXT_END();
}
// Create the storage
MEM_CONTEXT_BEGIN(storageMemContext)
MEM_CONTEXT_NEW_BEGIN("Storage")
{
result = (Storage *)memNew(sizeof(Storage));
result->path = strDup(path);
result->mode = mode;
result->bufferSize = bufferSize;
result->pathExpressionFunction = pathExpressionFunction;
this = (Storage *)memNew(sizeof(Storage));
this->memContext = MEM_CONTEXT_NEW();
this->path = strDup(path);
this->modeFile = param.modeFile == 0 ? STORAGE_FILE_MODE_DEFAULT : param.modeFile;
this->modePath = param.modePath == 0 ? STORAGE_PATH_MODE_DEFAULT : param.modePath;
this->bufferSize = param.bufferSize == 0 ? STORAGE_BUFFER_SIZE_DEFAULT : param.bufferSize;
this->write = param.write;
this->pathExpressionFunction = param.pathExpressionFunction;
}
MEM_CONTEXT_END();
MEM_CONTEXT_NEW_END();
return this;
}
/***********************************************************************************************************************************
Does a file/path exist?
***********************************************************************************************************************************/
bool
storageExists(const Storage *this, const String *pathExp, StorageExistsParam param)
{
bool result = false;
// Timeout can't be negative
ASSERT_DEBUG(param.timeout >= 0);
MEM_CONTEXT_TEMP_BEGIN()
{
// Build the path
String *path = storagePathNP(this, pathExp);
// Create Wait object of timeout > 0
Wait *wait = param.timeout != 0 ? waitNew(param.timeout) : NULL;
// Attempt to stat the file to determine if it exists
struct stat statFile;
do
{
// Any error other than entry not found should be reported
if (stat(strPtr(path), &statFile) == -1)
{
if (errno != ENOENT)
THROW_SYS_ERROR(FileOpenError, "unable to stat '%s'", strPtr(path));
}
// Else found
else
result = true;
}
while (!result && wait != NULL && waitMore(wait));
}
MEM_CONTEXT_TEMP_END();
return result;
}
@ -69,26 +121,14 @@ storageNew(const String *path, int mode, size_t bufferSize, StoragePathExpressio
Read from storage into a buffer
***********************************************************************************************************************************/
Buffer *
storageGet(const Storage *storage, const String *fileExp, bool ignoreMissing)
storageGet(const StorageFile *file)
{
Buffer volatile *result = NULL;
volatile int fileHandle = -1;
String *file = NULL;
TRY_BEGIN()
// Nothing to do unless a file was passed
if (file != NULL)
{
// Build the path
file = storagePath(storage, fileExp);
// Open the file and handle errors
fileHandle = open(strPtr(file), O_RDONLY, storage->mode);
if (fileHandle == -1)
{
if (!ignoreMissing || errno != ENOENT)
THROW_SYS_ERROR(FileOpenError, "unable to open '%s' for read", strPtr(file));
}
else
TRY_BEGIN()
{
// Create result buffer with buffer size
ssize_t actualBytes = 0;
@ -98,17 +138,18 @@ storageGet(const Storage *storage, const String *fileExp, bool ignoreMissing)
{
// Allocate the buffer before first read
if (result == NULL)
result = bufNew(storage->bufferSize);
result = bufNew(storageFileStorage(file)->bufferSize);
// Grow the buffer on subsequent reads
else
bufResize((Buffer *)result, bufSize((Buffer *)result) + (size_t)storage->bufferSize);
bufResize((Buffer *)result, bufSize((Buffer *)result) + (size_t)storageFileStorage(file)->bufferSize);
// Read and handle errors
actualBytes = read(fileHandle, bufPtr((Buffer *)result) + totalBytes, storage->bufferSize);
actualBytes = read(
STORAGE_DATA(file)->handle, bufPtr((Buffer *)result) + totalBytes, storageFileStorage(file)->bufferSize);
// Error occurred during write
if (actualBytes == -1)
THROW_SYS_ERROR(FileReadError, "unable to read '%s'", strPtr(file));
THROW_SYS_ERROR(FileReadError, "unable to read '%s'", strPtr(storageFileName(file)));
// Track total bytes read
totalBytes += (size_t)actualBytes;
@ -118,24 +159,20 @@ storageGet(const Storage *storage, const String *fileExp, bool ignoreMissing)
// Resize buffer to total bytes read
bufResize((Buffer *)result, totalBytes);
}
}
CATCH_ANY()
{
// Free buffer on error if it was allocated
bufFree((Buffer *)result);
CATCH_ANY()
{
// Free buffer on error if it was allocated
bufFree((Buffer *)result);
RETHROW();
RETHROW();
}
FINALLY()
{
close(STORAGE_DATA(file)->handle);
storageFileFree(file);
}
TRY_END();
}
FINALLY()
{
// Close file
if (fileHandle != -1)
close(fileHandle);
// Free file name
strFree(file);
}
TRY_END();
return (Buffer *)result;
}
@ -144,7 +181,7 @@ storageGet(const Storage *storage, const String *fileExp, bool ignoreMissing)
Get a list of files from a directory
***********************************************************************************************************************************/
StringList *
storageList(const Storage *storage, const String *pathExp, const String *expression, bool ignoreMissing)
storageList(const Storage *this, const String *pathExp, StorageListParam param)
{
StringList *result = NULL;
@ -155,7 +192,7 @@ storageList(const Storage *storage, const String *pathExp, const String *express
TRY_BEGIN()
{
// Build the path
path = storagePath(storage, pathExp);
path = storagePathNP(this, pathExp);
// Open the directory for read
dir = opendir(strPtr(path));
@ -163,14 +200,14 @@ storageList(const Storage *storage, const String *pathExp, const String *express
// If the directory could not be opened process errors but ignore missing directories when specified
if (!dir)
{
if (!ignoreMissing || errno != ENOENT)
if (param.errorOnMissing || errno != ENOENT)
THROW_SYS_ERROR(PathOpenError, "unable to open directory '%s' for read", strPtr(path));
}
else
{
// Prepare regexp if an expression was passed
if (expression != NULL)
regExp = regExpNew(expression);
if (param.expression != NULL)
regExp = regExpNew(param.expression);
// Create the string list now that we know the directory is valid
result = strLstNew();
@ -214,18 +251,106 @@ storageList(const Storage *storage, const String *pathExp, const String *express
return result;
}
/***********************************************************************************************************************************
Open a file for reading
***********************************************************************************************************************************/
StorageFile *
storageOpenRead(const Storage *this, const String *fileExp, StorageOpenReadParam param)
{
StorageFile *result = NULL;
MEM_CONTEXT_NEW_BEGIN("StorageFileRead")
{
String *fileName = storagePathNP(this, fileExp);
int fileHandle;
// Open the file and handle errors
fileHandle = open(strPtr(fileName), O_RDONLY, 0);
if (fileHandle == -1)
{
// Error unless ignore missing is specified
if (!param.ignoreMissing || errno != ENOENT)
THROW_SYS_ERROR(FileOpenError, "unable to open '%s' for read", strPtr(fileName));
// Free mem contexts if missing files are ignored
memContextSwitch(MEM_CONTEXT_OLD());
memContextFree(MEM_CONTEXT_NEW());
}
else
{
// Create the storage file and data
StorageFileDataPosix *data = NULL;
MEM_CONTEXT_NEW_BEGIN("StorageFileReadDataPosix")
{
data = memNew(sizeof(StorageFileDataPosix));
data->memContext = MEM_CONTEXT_NEW();
data->handle = fileHandle;
}
MEM_CONTEXT_NEW_END();
result = storageFileNew(this, fileName, storageFileTypeRead, data);
}
}
MEM_CONTEXT_NEW_END();
return result;
}
/***********************************************************************************************************************************
Open a file for writing
***********************************************************************************************************************************/
StorageFile *
storageOpenWrite(const Storage *this, const String *fileExp, StorageOpenWriteParam param)
{
ASSERT_STORAGE_ALLOWS_WRITE();
StorageFile *result = NULL;
MEM_CONTEXT_NEW_BEGIN("StorageFileWrite")
{
String *fileName = storagePathNP(this, fileExp);
int fileHandle;
// Open the file and handle errors
fileHandle = open(
strPtr(fileName), O_CREAT | O_TRUNC | O_WRONLY, param.mode == 0 ? this->modeFile : param.mode);
if (fileHandle == -1)
THROW_SYS_ERROR(FileOpenError, "unable to open '%s' for write", strPtr(fileName));
// Create the storage file and data
StorageFileDataPosix *data = NULL;
MEM_CONTEXT_NEW_BEGIN("StorageFileReadDataPosix")
{
data = memNew(sizeof(StorageFileDataPosix));
data->memContext = MEM_CONTEXT_NEW();
data->handle = fileHandle;
}
MEM_CONTEXT_NEW_END();
result = storageFileNew(this, fileName, storageFileTypeWrite, data);
}
MEM_CONTEXT_NEW_END();
return result;
}
/***********************************************************************************************************************************
Get the absolute path in the storage
***********************************************************************************************************************************/
String *
storagePath(const Storage *storage, const String *pathExp)
storagePath(const Storage *this, const String *pathExp)
{
String *result = NULL;
// If there there is no path expression then return the base storage path
if (pathExp == NULL)
{
result = strDup(storage->path);
result = strDup(this->path);
}
else
{
@ -233,10 +358,13 @@ storagePath(const Storage *storage, const String *pathExp)
if ((strPtr(pathExp))[0] == '/')
{
// Make sure the base storage path is contained within the path expression
if (!strEqZ(storage->path, "/"))
if (!strEqZ(this->path, "/"))
{
if (!strBeginsWith(pathExp, storage->path) || *(strPtr(pathExp) + strSize(storage->path)) != '/')
THROW(AssertError, "absolute path '%s' is not in base path '%s'", strPtr(pathExp), strPtr(storage->path));
if (!strBeginsWith(pathExp, this->path) ||
!(strSize(pathExp) == strSize(this->path) || *(strPtr(pathExp) + strSize(this->path)) == '/'))
{
THROW(AssertError, "absolute path '%s' is not in base path '%s'", strPtr(pathExp), strPtr(this->path));
}
}
result = strDup(pathExp);
@ -250,7 +378,7 @@ storagePath(const Storage *storage, const String *pathExp)
// Check if there is a path expression that needs to be evaluated
if ((strPtr(pathExp))[0] == '<')
{
if (storage->pathExpressionFunction == NULL)
if (this->pathExpressionFunction == NULL)
THROW(AssertError, "expression '%s' not valid without callback function", strPtr(pathExp));
// Get position of the expression end
@ -280,7 +408,7 @@ storagePath(const Storage *storage, const String *pathExp)
}
// Evaluate the path
pathEvaluated = storage->pathExpressionFunction(expression, path);
pathEvaluated = this->pathExpressionFunction(expression, path);
// Evaluated path cannot be NULL
if (pathEvaluated == NULL)
@ -294,10 +422,10 @@ storagePath(const Storage *storage, const String *pathExp)
strFree(path);
}
if (strEqZ(storage->path, "/"))
if (strEqZ(this->path, "/"))
result = strNewFmt("/%s", strPtr(pathExp));
else
result = strNewFmt("%s/%s", strPtr(storage->path), strPtr(pathExp));
result = strNewFmt("%s/%s", strPtr(this->path), strPtr(pathExp));
strFree(pathEvaluated);
}
@ -307,57 +435,128 @@ storagePath(const Storage *storage, const String *pathExp)
}
/***********************************************************************************************************************************
Check for errors on write. This is a separate function for testing purposes.
Create a path
***********************************************************************************************************************************/
static void
storageWriteError(ssize_t actualBytes, size_t expectedBytes, const String *file)
void
storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param)
{
// Error occurred during write
if (actualBytes == -1)
{
int errNo = errno;
THROW(FileWriteError, "unable to write '%s': %s", strPtr(file), strerror(errNo));
}
ASSERT_STORAGE_ALLOWS_WRITE();
// Make sure that all expected bytes were written. Cast to unsigned since we have already tested for -1.
if ((size_t)actualBytes != expectedBytes)
THROW(FileWriteError, "only wrote %lu byte(s) to '%s' but %lu byte(s) expected", actualBytes, strPtr(file), expectedBytes);
// It doesn't make sense to combine these parameters because if we are creating missing parent paths why error when they exist?
// If this somehow wasn't caught in testing, the worst case is that the path would not be created and an error would be thrown.
ASSERT_DEBUG(!(param.noParentCreate && param.errorOnExists));
MEM_CONTEXT_TEMP_BEGIN()
{
// Build the path
String *path = storagePathNP(this, pathExp);
// Attempt to create the directory
if (mkdir(strPtr(path), param.mode != 0 ? param.mode : STORAGE_PATH_MODE_DEFAULT) == -1)
{
// If the parent path does not exist then create it if allowed
if (errno == ENOENT && !param.noParentCreate)
{
storagePathCreate(this, strPath(path), param);
storagePathCreate(this, path, param);
}
// Ignore path exists if allowed
else if (errno != EEXIST || param.errorOnExists)
THROW_SYS_ERROR(PathCreateError, "unable to create path '%s'", strPtr(path));
}
}
MEM_CONTEXT_TEMP_END();
}
/***********************************************************************************************************************************
Write a buffer to storage
***********************************************************************************************************************************/
void
storagePut(const Storage *storage, const String *fileExp, const Buffer *buffer)
storagePut(const StorageFile *file, const Buffer *buffer)
{
volatile int fileHandle = -1;
String *file = NULL;
// Write data if buffer is not null. Otherwise, an empty file is expected.
if (buffer != NULL)
{
TRY_BEGIN()
{
if (write(STORAGE_DATA(file)->handle, bufPtr(buffer), bufSize(buffer)) != (ssize_t)bufSize(buffer))
THROW_SYS_ERROR(FileWriteError, "unable to write '%s'", strPtr(storageFileName(file)));
}
FINALLY()
{
close(STORAGE_DATA(file)->handle);
storageFileFree(file);
}
TRY_END();
}
}
TRY_BEGIN()
/***********************************************************************************************************************************
Remove a file
***********************************************************************************************************************************/
void
storageRemove(const Storage *this, const String *pathExp, StorageRemoveParam param)
{
ASSERT_STORAGE_ALLOWS_WRITE();
MEM_CONTEXT_TEMP_BEGIN()
{
// Build the path
file = storagePath(storage, fileExp);
String *file = storagePathNP(this, pathExp);
// Open the file and handle errors
fileHandle = open(strPtr(file), O_CREAT | O_TRUNC | O_WRONLY, storage->mode);
if (fileHandle == -1)
// Attempt to unlink the file
if (unlink(strPtr(file)) == -1)
{
int errNo = errno;
THROW(FileOpenError, "unable to open '%s' for write: %s", strPtr(file), strerror(errNo));
if (param.errorOnMissing || errno != ENOENT)
THROW_SYS_ERROR(FileRemoveError, "unable to remove '%s'", strPtr(file));
}
// Write data if buffer is not null. Otherwise, and empty file is expected.
if (buffer != NULL)
storageWriteError(write(fileHandle, bufPtr(buffer), bufSize(buffer)), bufSize(buffer), file);
}
FINALLY()
{
if (fileHandle != -1)
close(fileHandle);
// Free file name
strFree(file);
}
TRY_END();
MEM_CONTEXT_TEMP_END();
}
/***********************************************************************************************************************************
Stat a file
***********************************************************************************************************************************/
StorageStat *
storageStat(const Storage *this, const String *pathExp, StorageStatParam param)
{
StorageStat *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Build the path
String *path = storagePathNP(this, pathExp);
// Attempt to stat the file
struct stat statFile;
if (stat(strPtr(path), &statFile) == -1)
{
if (errno != ENOENT || !param.ignoreMissing)
THROW_SYS_ERROR(FileOpenError, "unable to stat '%s'", strPtr(path));
}
// Else set stats
else
{
memContextSwitch(MEM_CONTEXT_OLD());
result = memNew(sizeof(StorageStat));
result->mode = statFile.st_mode & 0777;
memContextSwitch(MEM_CONTEXT_TEMP());
}
}
MEM_CONTEXT_TEMP_END();
return result;
}
/***********************************************************************************************************************************
Free storage
***********************************************************************************************************************************/
void
storageFree(const Storage *this)
{
if (this != NULL)
memContextFree(this->memContext);
}

184
src/storage/storage.h
View File

@ -1,12 +1,22 @@
/***********************************************************************************************************************************
Storage Manager
***********************************************************************************************************************************/
#ifndef STORAGE_LOCAL_H
#define STORAGE_LOCAL_H
#ifndef STORAGE_STORAGE_H
#define STORAGE_STORAGE_H
#include <sys/types.h>
#include "common/type/buffer.h"
#include "common/type/string.h"
/***********************************************************************************************************************************
Default buffer size
Generally buffer size should be pulled out of options but some storage is created without access to options. In those cases we'll
assume they are not doing any heavy lifting and have a moderate default.
***********************************************************************************************************************************/
#define STORAGE_BUFFER_SIZE_DEFAULT (64 * 1024)
/***********************************************************************************************************************************
Default file and path modes
***********************************************************************************************************************************/
@ -18,19 +28,175 @@ Storage object
***********************************************************************************************************************************/
typedef struct Storage Storage;
#include "storage/file.h"
/***********************************************************************************************************************************
Path expression callback function type - used to modify paths base on expressions enclosed in <>
Path expression callback function type - used to modify paths based on expressions enclosed in <>
***********************************************************************************************************************************/
typedef String *(*StoragePathExpressionCallback)(const String *expression, const String *path);
/***********************************************************************************************************************************
Functions
storageNew
***********************************************************************************************************************************/
Storage *storageNew(const String *path, int mode, size_t bufferSize, StoragePathExpressionCallback pathExpressionFunction);
typedef struct StorageNewParam
{
mode_t modeFile;
mode_t modePath;
size_t bufferSize;
bool write;
StoragePathExpressionCallback pathExpressionFunction;
} StorageNewParam;
Buffer *storageGet(const Storage *storage, const String *fileExp, bool ignoreMissing);
StringList *storageList(const Storage *storage, const String *pathExp, const String *expression, bool ignoreMissing);
String *storagePath(const Storage *storage, const String *pathExp);
void storagePut(const Storage *storage, const String *fileExp, const Buffer *buffer);
#define storageNewP(path, ...) \
storageNew(path, (StorageNewParam){__VA_ARGS__})
#define storageNewNP(path) \
storageNew(path, (StorageNewParam){0})
Storage *storageNew(const String *path, StorageNewParam param);
/***********************************************************************************************************************************
storageExists
***********************************************************************************************************************************/
typedef struct StorageExistsParam
{
double timeout;
} StorageExistsParam;
#define storageExistsP(this, pathExp, ...) \
storageExists(this, pathExp, (StorageExistsParam){__VA_ARGS__})
#define storageExistsNP(this, pathExp) \
storageExists(this, pathExp, (StorageExistsParam){0})
bool storageExists(const Storage *this, const String *pathExp, StorageExistsParam param);
/***********************************************************************************************************************************
storageGet
***********************************************************************************************************************************/
#define storageGetNP(file) \
storageGet(file)
Buffer *storageGet(const StorageFile *file);
/***********************************************************************************************************************************
storageList
***********************************************************************************************************************************/
typedef struct StorageListParam
{
bool errorOnMissing;
const String *expression;
} StorageListParam;
#define storageListP(this, pathExp, ...) \
storageList(this, pathExp, (StorageListParam){__VA_ARGS__})
#define storageListNP(this, pathExp) \
storageList(this, pathExp, (StorageListParam){0})
StringList *storageList(const Storage *this, const String *pathExp, StorageListParam param);
/***********************************************************************************************************************************
storageOpenRead
***********************************************************************************************************************************/
typedef struct StorageOpenReadParam
{
bool ignoreMissing;
} StorageOpenReadParam;
#define storageOpenReadP(this, pathExp, ...) \
storageOpenRead(this, pathExp, (StorageOpenReadParam){__VA_ARGS__})
#define storageOpenReadNP(this, pathExp) \
storageOpenRead(this, pathExp, (StorageOpenReadParam){0})
StorageFile *storageOpenRead(const Storage *this, const String *fileExp, StorageOpenReadParam param);
/***********************************************************************************************************************************
storageOpenWrite
***********************************************************************************************************************************/
typedef struct StorageOpenWriteParam
{
mode_t mode;
} StorageOpenWriteParam;
#define storageOpenWriteP(this, pathExp, ...) \
storageOpenWrite(this, pathExp, (StorageOpenWriteParam){__VA_ARGS__})
#define storageOpenWriteNP(this, pathExp) \
storageOpenWrite(this, pathExp, (StorageOpenWriteParam){0})
StorageFile *storageOpenWrite(const Storage *this, const String *fileExp, StorageOpenWriteParam param);
/***********************************************************************************************************************************
storagePath
***********************************************************************************************************************************/
#define storagePathNP(this, pathExp) \
storagePath(this, pathExp)
String *storagePath(const Storage *this, const String *pathExp);
/***********************************************************************************************************************************
storagePathCreate
***********************************************************************************************************************************/
typedef struct StoragePathCreateParam
{
bool errorOnExists;
bool noParentCreate;
mode_t mode;
} StoragePathCreateParam;
#define storagePathCreateP(this, pathExp, ...) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){__VA_ARGS__})
#define storagePathCreateNP(this, pathExp) \
storagePathCreate(this, pathExp, (StoragePathCreateParam){0})
void storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param);
/***********************************************************************************************************************************
storagePut
***********************************************************************************************************************************/
#define storagePutNP(file, buffer) \
storagePut(file, buffer)
void storagePut(const StorageFile *file, const Buffer *buffer);
/***********************************************************************************************************************************
storageRemove
***********************************************************************************************************************************/
typedef struct StorageRemoveParam
{
bool errorOnMissing;
} StorageRemoveParam;
#define storageRemoveP(this, pathExp, ...) \
storageRemove(this, pathExp, (StorageRemoveParam){__VA_ARGS__})
#define storageRemoveNP(this, pathExp) \
storageRemove(this, pathExp, (StorageRemoveParam){0})
void storageRemove(const Storage *this, const String *fileExp, StorageRemoveParam param);
/***********************************************************************************************************************************
storageStat
***********************************************************************************************************************************/
typedef struct StorageStat
{
mode_t mode;
} StorageStat;
typedef struct StorageStatParam
{
bool ignoreMissing;
} StorageStatParam;
#define storageStatP(this, pathExp, ...) \
storageStat(this, pathExp, (StorageStatParam){__VA_ARGS__})
#define storageStatNP(this, pathExp) \
storageStat(this, pathExp, (StorageStatParam){0})
StorageStat *storageStat(const Storage *this, const String *pathExp, StorageStatParam param);
/***********************************************************************************************************************************
storageFree
***********************************************************************************************************************************/
#define storageFreeNP(this) \
storageFree(this)
void storageFree(const Storage *this);
#endif

14
test/lib/pgBackRestTest/Common/DefineTest.pm
View File

@ -614,9 +614,19 @@ my $oTestDef =
'Storage/Helper' => TESTDEF_COVERAGE_PARTIAL,
},
},
{
&TESTDEF_NAME => 'file',
&TESTDEF_TOTAL => 1,
&TESTDEF_C => true,
&TESTDEF_COVERAGE =>
{
'storage/file' => TESTDEF_COVERAGE_FULL,
},
},
{
&TESTDEF_NAME => 'storage',
&TESTDEF_TOTAL => 4,
&TESTDEF_TOTAL => 10,
&TESTDEF_C => true,
&TESTDEF_COVERAGE =>
@ -626,7 +636,7 @@ my $oTestDef =
},
{
&TESTDEF_NAME => 'helper',
&TESTDEF_TOTAL => 2,
&TESTDEF_TOTAL => 3,
&TESTDEF_C => true,
&TESTDEF_COVERAGE =>

3
test/lib/pgBackRestTest/Common/JobTest.pm
View File

@ -342,6 +342,9 @@ sub run
"CFLAGS=-I. -std=c99 -fPIC -g \\\n" .
" -Werror -Wfatal-errors -Wall -Wextra -Wwrite-strings -Wno-clobbered -Wswitch-enum -Wconversion \\\n" .
($self->{oTest}->{&TEST_VM} eq VM_U16 ? " -Wformat-signedness \\\n" : '') .
# This warning appears to be broken on U12 even though the functionality is fine
($self->{oTest}->{&TEST_VM} eq VM_U12 || $self->{oTest}->{&TEST_VM} eq VM_CO6 ?
" -Wno-missing-field-initializers \\\n" : '') .
($self->{oTest}->{&TEST_VM} ne VM_CO6 && $self->{oTest}->{&TEST_VM} ne VM_U12 ?
" -Wpedantic \\\n" : '') .
" -Wformat=2 -Wformat-nonliteral \\\n" .

8
test/src/common/logTest.c
View File

@ -41,7 +41,7 @@ After the comparison the log is cleared so the next result can be compared.
void
testLogResult(const char *expected)
{
String *actual = strTrim(strNewBuf(storageGet(storageLocal(), stdoutFile, false)));
String *actual = strTrim(strNewBuf(storageGetNP(storageOpenReadNP(storageLocal(), stdoutFile))));
if (!strEqZ(actual, expected))
THROW(AssertError, "\n\nexpected log:\n\n%s\n\nbut actual log was:\n\n%s\n\n", expected, strPtr(actual));
@ -58,7 +58,7 @@ After the comparison the log is cleared so the next result can be compared.
void
testLogErrResult(const char *expected)
{
String *actual = strTrim(strNewBuf(storageGet(storageLocal(), stderrFile, false)));
String *actual = strTrim(strNewBuf(storageGetNP(storageOpenReadNP(storageLocal(), stderrFile))));
if (!strEqZ(actual, expected))
THROW(AssertError, "\n\nexpected error log:\n\n%s\n\nbut actual error log was:\n\n%s\n\n", expected, strPtr(actual));
@ -73,12 +73,12 @@ Make sure nothing is left in the log after all tests have completed
void
testLogFinal()
{
String *actual = strTrim(strNewBuf(storageGet(storageLocal(), stdoutFile, false)));
String *actual = strTrim(strNewBuf(storageGetNP(storageOpenReadNP(storageLocal(), stdoutFile))));
if (!strEqZ(actual, ""))
THROW(AssertError, "\n\nexpected log to be empty but actual log was:\n\n%s\n\n", strPtr(actual));
actual = strTrim(strNewBuf(storageGet(storageLocal(), stderrFile, false)));
actual = strTrim(strNewBuf(storageGetNP(storageOpenReadNP(storageLocal(), stderrFile))));
if (!strEqZ(actual, ""))
THROW(AssertError, "\n\nexpected error log to be empty but actual error log was:\n\n%s\n\n", strPtr(actual));

48
test/src/module/archive/pushTest.c
View File

@ -38,49 +38,55 @@ testRun()
TEST_RESULT_BOOL(walStatus(segment, false), false, "status file not present");
// -------------------------------------------------------------------------------------------------------------------------
storagePut(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), bufNewStr(strNew(BOGUS_STR)));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
bufNewStr(strNew(BOGUS_STR)));
TEST_ERROR(walStatus(segment, false), FormatError, "000000010000000100000001.ok content must have at least two lines");
storagePut(
storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), bufNewStr(strNew(BOGUS_STR "\n")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
bufNewStr(strNew(BOGUS_STR "\n")));
TEST_ERROR(walStatus(segment, false), FormatError, "000000010000000100000001.ok message must be > 0");
storagePut(
storageSpool(),
strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), bufNewStr(strNew(BOGUS_STR "\nmessage")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
bufNewStr(strNew(BOGUS_STR "\nmessage")));
TEST_ERROR(walStatus(segment, false), FormatError, "unable to convert str 'BOGUS' to int");
storagePut(
storageSpool(),
strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), bufNewStr(strNew("0\nwarning")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
bufNewStr(strNew("0\nwarning")));
TEST_RESULT_BOOL(walStatus(segment, false), true, "ok file with warning");
testLogResult("P00 WARN: warning");
storagePut(
storageSpool(),
strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), bufNewStr(strNew("25\nerror")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
bufNewStr(strNew("25\nerror")));
TEST_RESULT_BOOL(walStatus(segment, false), true, "error status renamed to ok");
testLogResult(
"P00 WARN: WAL segment '000000010000000100000001' was not pushed due to error [25] and was manually skipped: error");
// -------------------------------------------------------------------------------------------------------------------------
storagePut(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment)), bufNewStr(strNew("")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment))),
bufNewStr(strNew("")));
TEST_ERROR(
walStatus(segment, false), AssertError,
strPtr(
strNewFmt(
"multiple status files found in '%s/archive/db/out' for WAL segment '000000010000000100000001'", testPath())));
unlink(strPtr(storagePath(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)))));
unlink(strPtr(storagePathNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)))));
TEST_ERROR(walStatus(segment, true), AssertError, "status file '000000010000000100000001.error' has no content");
storagePut(
storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment)), bufNewStr(strNew("25\nmessage")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment))),
bufNewStr(strNew("25\nmessage")));
TEST_ERROR(walStatus(segment, true), AssertError, "message");
TEST_RESULT_BOOL(walStatus(segment, false), false, "suppress error");
unlink(strPtr(storagePath(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment)))));
unlink(strPtr(storagePathNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment)))));
}
// *****************************************************************************************************************************
@ -136,12 +142,12 @@ testRun()
// Write out a bogus .error file to make sure it is ignored on the first loop
// -------------------------------------------------------------------------------------------------------------------------
String *errorFile = storagePath(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.error"));
String *errorFile = storagePathNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.error"));
mkdir(strPtr(strNewFmt("%s/archive", testPath())), 0750);
mkdir(strPtr(strNewFmt("%s/archive/db", testPath())), 0750);
mkdir(strPtr(strNewFmt("%s/archive/db/out", testPath())), 0750);
storagePut(storageSpool(), errorFile, bufNewStr(strNew("25\n" BOGUS_STR)));
storagePutNP(storageOpenWriteNP(storageSpool(), errorFile), bufNewStr(strNew("25\n" BOGUS_STR)));
TEST_ERROR(cmdArchivePush(), AssertError, BOGUS_STR);
@ -149,7 +155,9 @@ testRun()
// Write out a valid ok file and test for success
// -------------------------------------------------------------------------------------------------------------------------
storagePut(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok"), bufNewStr(strNew("")));
storagePutNP(
storageOpenWriteNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok")),
bufNewStr(strNew("")));
TEST_RESULT_VOID(cmdArchivePush(), "successful push");
testLogResult("P00 INFO: pushed WAL segment 000000010000000100000001 asynchronously");

2
test/src/module/common/iniTest.c
View File

@ -88,7 +88,7 @@ testRun()
"\n"
);
TEST_RESULT_VOID(storagePut(storageLocal(), fileName, bufNewStr(content)), "put ini to file");
TEST_RESULT_VOID(storagePutNP(storageOpenWriteNP(storageLocalWrite(), fileName), bufNewStr(content)), "put ini to file");
TEST_RESULT_VOID(iniLoad(ini, fileName), "load ini from file");
TEST_RESULT_STR(strPtr(varStr(iniGet(ini, strNew("global"), strNew("compress")))), "y", "get compress");

8
test/src/module/common/logTest.c
View File

@ -125,10 +125,10 @@ testRun()
logFileSet("/" BOGUS_STR);
// Check stdout
Storage *storage = storageNew(strNew(testPath()), 0750, 65536, NULL);
Storage *storage = storageNewNP(strNew(testPath()));
TEST_RESULT_STR(
strPtr(strNewBuf(storageGet(storage, stdoutFile, false))),
strPtr(strNewBuf(storageGetNP(storageOpenReadNP(storage, stdoutFile)))),
"P00 WARN: format 99\n"
"P00 ERROR: [026]: message\n"
"P00 ERROR: [026]: message1\n"
@ -137,7 +137,7 @@ testRun()
// Check stderr
TEST_RESULT_STR(
strPtr(strNewBuf(storageGet(storage, stderrFile, false))),
strPtr(strNewBuf(storageGetNP(storageOpenReadNP(storage, stderrFile)))),
"DEBUG: test.c:test_func(): message\n"
"INFO: info message\n"
"WARN: unable to open log file '/BOGUS': Permission denied\n"
@ -146,7 +146,7 @@ testRun()
// Check file
TEST_RESULT_STR(
strPtr(strNewBuf(storageGet(storage, fileFile, false))),
strPtr(strNewBuf(storageGetNP(storageOpenReadNP(storage, fileFile)))),
"-------------------PROCESS START-------------------\n"
"P00 DEBUG: test.c:test_func(): message\n"
"\n"

2
test/src/module/common/timeTest.c
View File

@ -25,7 +25,7 @@ testRun()
TimeMSec end = timeMSec();
// Check bounds for time slept (within a range of .1 seconds)
TEST_RESULT_BOOL(end - begin > (TimeMSec)1400, true, "lower range check");
TEST_RESULT_BOOL(end - begin >= (TimeMSec)1400, true, "lower range check");
TEST_RESULT_BOOL(end - begin < (TimeMSec)1500, true, "upper range check");
}
}

7
test/src/module/common/typeStringTest.c
View File

@ -39,12 +39,17 @@ testRun()
}
// *****************************************************************************************************************************
if (testBegin("strBase()"))
if (testBegin("strBase() and strPath()"))
{
TEST_RESULT_STR(strPtr(strBase(strNew(""))), "", "empty string");
TEST_RESULT_STR(strPtr(strBase(strNew("/"))), "", "/ only");
TEST_RESULT_STR(strPtr(strBase(strNew("/file"))), "file", "root file");
TEST_RESULT_STR(strPtr(strBase(strNew("/dir1/dir2/file"))), "file", "subdirectory file");
TEST_RESULT_STR(strPtr(strPath(strNew(""))), "", "empty string");
TEST_RESULT_STR(strPtr(strPath(strNew("/"))), "/", "/ only");
TEST_RESULT_STR(strPtr(strPath(strNew("/file"))), "/", "root path");
TEST_RESULT_STR(strPtr(strPath(strNew("/dir1/dir2/file"))), "/dir1/dir2", "subdirectory file");
}
// *****************************************************************************************************************************

16
test/src/module/config/parseTest.c
View File

@ -323,7 +323,7 @@ testRun()
strLstAdd(argList, strNewFmt("--config=%s", strPtr(configFile)));
strLstAdd(argList, strNew(TEST_COMMAND_BACKUP));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global]\n"
"compress=bogus\n"
)));
@ -338,7 +338,7 @@ testRun()
strLstAdd(argList, strNewFmt("--config=%s", strPtr(configFile)));
strLstAdd(argList, strNew(TEST_COMMAND_BACKUP));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global]\n"
"compress=\n"
)));
@ -354,7 +354,7 @@ testRun()
strLstAdd(argList, strNewFmt("--config=%s", strPtr(configFile)));
strLstAdd(argList, strNew(TEST_COMMAND_BACKUP));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[db]\n"
"pg1-path=/path/to/db\n"
"db-path=/also/path/to/db\n"
@ -371,7 +371,7 @@ testRun()
strLstAdd(argList, strNewFmt("--config=%s", strPtr(configFile)));
strLstAdd(argList, strNew(TEST_COMMAND_BACKUP));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[db]\n"
"pg1-path=/path/to/db\n"
"pg1-path=/also/path/to/db\n"
@ -473,7 +473,7 @@ testRun()
strLstAdd(argList, strNew("--reset-backup-standby"));
strLstAdd(argList, strNew(TEST_COMMAND_BACKUP));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global]\n"
"compress-level=3\n"
"spool-path=/path/to/spool\n"
@ -536,7 +536,7 @@ testRun()
strLstAdd(argList, strNew("--archive-queue-max=4503599627370496"));
strLstAdd(argList, strNew("archive-push"));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global]\n"
"spool-path=/path/to/spool\n"
)));
@ -607,7 +607,7 @@ testRun()
strLstAdd(argList, strNew("--stanza=db"));
strLstAdd(argList, strNew(TEST_COMMAND_RESTORE));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global:restore]\n"
"recovery-option=f=g\n"
"recovery-option=hijk=l\n"
@ -629,7 +629,7 @@ testRun()
strLstAdd(argList, strNewFmt("--config=%s", strPtr(configFile)));
strLstAdd(argList, strNew("info"));
storagePut(storageLocal(), configFile, bufNewStr(strNew(
storagePutNP(storageOpenWriteNP(storageLocalWrite(), configFile), bufNewStr(strNew(
"[global]\n"
"repo1-path=/path/to/repo\n"
"\n"

4
test/src/module/help/helpTest.c
View File

@ -290,7 +290,7 @@ testRun()
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
Storage *storage = storageNew(strNew(testPath()), 0750, 65536, NULL);
TEST_RESULT_STR(strPtr(strNewBuf(storageGet(storage, stdoutFile, false))), generalHelp, " check text");
Storage *storage = storageNewNP(strNew(testPath()));
TEST_RESULT_STR(strPtr(strNewBuf(storageGetNP(storageOpenReadNP(storage, stdoutFile)))), generalHelp, " check text");
}
}

33
test/src/module/storage/fileTest.c Normal file
View File

@ -0,0 +1,33 @@
/***********************************************************************************************************************************
Test Storage File
***********************************************************************************************************************************/
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
testRun()
{
// *****************************************************************************************************************************
if (testBegin("storageFileNew() and storageFileFree()"))
{
MemContext *memContext = NULL;
StorageFile *file = NULL;
MEM_CONTEXT_NEW_BEGIN("TestFile")
{
memContext = MEM_CONTEXT_NEW();
TEST_ASSIGN(file, storageFileNew((Storage *)1, strNew("file"), storageFileTypeRead, (void *)2), "new file");
}
MEM_CONTEXT_NEW_END();
TEST_RESULT_PTR(storageFileData(file), (void *)2, " check data");
TEST_RESULT_PTR(file->memContext, memContext, " check mem context");
TEST_RESULT_STR(strPtr(storageFileName(file)), "file", " check name");
TEST_RESULT_INT(file->type, storageFileTypeRead, " check type");
TEST_RESULT_PTR(storageFileStorage(file), (Storage *)1, " check storage");
TEST_RESULT_VOID(storageFileFree(file), "free file");
TEST_RESULT_VOID(storageFileFree(NULL), "free null file");
}
}

30
test/src/module/storage/helperTest.c
View File

@ -8,6 +8,8 @@ Test Run
void
testRun()
{
String *writeFile = strNewFmt("%s/writefile", testPath());
// -----------------------------------------------------------------------------------------------------------------------------
if (testBegin("storageLocal()"))
{
@ -18,7 +20,23 @@ testRun()
TEST_RESULT_PTR(storageLocalData, storage, "local storage cached");
TEST_RESULT_PTR(storageLocal(), storage, "get cached storage");
TEST_RESULT_STR(strPtr(storagePath(storage, NULL)), "/", "check base path");
TEST_RESULT_STR(strPtr(storagePathNP(storage, NULL)), "/", "check base path");
TEST_ERROR(storageOpenWriteNP(storage, writeFile), AssertError, "assertion 'this->write == true' failed");
}
// -----------------------------------------------------------------------------------------------------------------------------
if (testBegin("storageLocalWrite()"))
{
const Storage *storage = NULL;
TEST_RESULT_PTR(storageLocalWriteData, NULL, "local storage not cached");
TEST_ASSIGN(storage, storageLocalWrite(), "new storage");
TEST_RESULT_PTR(storageLocalWriteData, storage, "local storage cached");
TEST_RESULT_PTR(storageLocalWrite(), storage, "get cached storage");
TEST_RESULT_STR(strPtr(storagePathNP(storage, NULL)), "/", "check base path");
TEST_RESULT_VOID(storageOpenWriteNP(storage, writeFile), "writes are allowed");
}
// -----------------------------------------------------------------------------------------------------------------------------
@ -37,14 +55,16 @@ testRun()
TEST_RESULT_PTR(storageSpoolData, storage, "storage cached");
TEST_RESULT_PTR(storageSpool(), storage, "get cached storage");
TEST_RESULT_STR(strPtr(storagePath(storage, NULL)), testPath(), "check base path");
TEST_RESULT_STR(strPtr(storagePathNP(storage, NULL)), testPath(), "check base path");
TEST_RESULT_STR(
strPtr(storagePath(storage, strNew(STORAGE_SPOOL_ARCHIVE_OUT))), strPtr(strNewFmt("%s/archive/db/out", testPath())),
strPtr(storagePathNP(storage, strNew(STORAGE_SPOOL_ARCHIVE_OUT))), strPtr(strNewFmt("%s/archive/db/out", testPath())),
"check spool out path");
TEST_RESULT_STR(
strPtr(storagePath(storage, strNewFmt("%s/%s", STORAGE_SPOOL_ARCHIVE_OUT, "file.ext"))),
strPtr(storagePathNP(storage, strNewFmt("%s/%s", STORAGE_SPOOL_ARCHIVE_OUT, "file.ext"))),
strPtr(strNewFmt("%s/archive/db/out/file.ext", testPath())), "check spool out path");
TEST_ERROR(storagePath(storage, strNew("<" BOGUS_STR ">")), AssertError, "invalid expression '<BOGUS>'");
TEST_ERROR(storagePathNP(storage, strNew("<" BOGUS_STR ">")), AssertError, "invalid expression '<BOGUS>'");
TEST_RESULT_VOID(storageOpenWriteNP(storage, writeFile), "writes are allowed");
}
}

334
test/src/module/storage/storageTest.c
View File

@ -1,6 +1,8 @@
/***********************************************************************************************************************************
Test Storage Manager
***********************************************************************************************************************************/
#include "common/time.h"
#include "storage/file.h"
/***********************************************************************************************************************************
Test function for path expression
@ -24,146 +26,264 @@ Test Run
void
testRun()
{
// Create default storage object for testing
Storage *storageTest = storageNewP(strNew(testPath()), .write = true);
// Create a directory and file that cannot be accessed to test permissions errors
String *fileNoPerm = strNewFmt("%s/noperm/noperm", testPath());
String *pathNoPerm = strPath(fileNoPerm);
TEST_RESULT_INT(
system(
strPtr(strNewFmt("sudo mkdir -m 700 %s && sudo touch %s && sudo chmod 600 %s", strPtr(pathNoPerm), strPtr(fileNoPerm),
strPtr(fileNoPerm)))),
0, "create no perm file");
// *****************************************************************************************************************************
if (testBegin("storageNew()"))
if (testBegin("storageNew() and storageFree()"))
{
Storage *storage = NULL;
Storage *storageTest = NULL;
TEST_ERROR(storageNew(NULL, 0750, 65536, NULL), AssertError, "storage base path cannot be null");
TEST_ERROR(storageNewNP(NULL), AssertError, "storage base path cannot be null");
TEST_ASSIGN(storage, storageNew(strNew("/"), 0750, 65536, NULL), "new storage");
TEST_RESULT_INT(storage->mode, 0750, "check mode");
TEST_RESULT_INT(storage->bufferSize, 65536, "check buffer size");
TEST_RESULT_STR(strPtr(storage->path), "/", "check path");
TEST_ASSIGN(storageTest, storageNewNP(strNew("/")), "new storage (defaults)");
TEST_RESULT_STR(strPtr(storageTest->path), "/", " check path");
TEST_RESULT_INT(storageTest->modeFile, 0640, " check file mode");
TEST_RESULT_INT(storageTest->modePath, 0750, " check path mode");
TEST_RESULT_INT(storageTest->bufferSize, 65536, " check buffer size");
TEST_RESULT_BOOL(storageTest->write, false, " check write");
TEST_RESULT_BOOL(storageTest->pathExpressionFunction == NULL, true, " check expression function is not set");
TEST_ASSIGN(
storageTest,
storageNewP(
strNew("/path/to"), .modeFile = 0600, .modePath = 0700, .bufferSize = 8192, .write = true,
.pathExpressionFunction = storageTestPathExpression),
"new storage (non-default)");
TEST_RESULT_STR(strPtr(storageTest->path), "/path/to", " check path");
TEST_RESULT_INT(storageTest->modeFile, 0600, " check file mode");
TEST_RESULT_INT(storageTest->modePath, 0700, " check path mode");
TEST_RESULT_INT(storageTest->bufferSize, 8192, " check buffer size");
TEST_RESULT_BOOL(storageTest->write, true, " check write");
TEST_RESULT_BOOL(storageTest->pathExpressionFunction != NULL, true, " check expression function is set");
TEST_RESULT_VOID(storageFree(storageTest), "free storage");
TEST_RESULT_VOID(storageFree(NULL), "free null storage");
}
// *****************************************************************************************************************************
if (testBegin("storagePath()"))
if (testBegin("storageExists()"))
{
Storage *storage = NULL;
TEST_ASSIGN(storage, storageNew(strNew("/"), 0750, 65536, NULL), "new storage /");
TEST_RESULT_STR(strPtr(storagePath(storage, NULL)), "/", " root dir");
TEST_RESULT_STR(strPtr(storagePath(storage, strNew("/"))), "/", " same as root dir");
TEST_RESULT_STR(strPtr(storagePath(storage, strNew("subdir"))), "/subdir", " simple subdir");
TEST_ERROR(storagePath(storage, strNew("<TEST>")), AssertError, "expression '<TEST>' not valid without callback function");
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_BOOL(storageExistsNP(storageTest, strNew("missing")), false, "file does not exist");
TEST_RESULT_BOOL(storageExistsP(storageTest, strNew("missing"), .timeout = .1), false, "file does not exist");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(
storage, storageNew(strNew("/path/to"), 0750, 65536, (StoragePathExpressionCallback)storageTestPathExpression),
"new storage /path/to with expression");
TEST_RESULT_STR(strPtr(storagePath(storage, NULL)), "/path/to", " root dir");
TEST_RESULT_STR(strPtr(storagePath(storage, strNew("is/a/subdir"))), "/path/to/is/a/subdir", " subdir");
TEST_ERROR(storagePath(storage, strNew("/bogus")), AssertError, "absolute path '/bogus' is not in base path '/path/to'");
TEST_ERROR(
storagePath(storage, strNew("/path/toot")), AssertError, "absolute path '/path/toot' is not in base path '/path/to'");
storageExistsNP(storageTest, fileNoPerm), FileOpenError,
strPtr(strNewFmt("unable to stat '%s': [13] Permission denied", strPtr(fileNoPerm))));
TEST_ERROR(storagePath(storage, strNew("<TEST")), AssertError, "end > not found in path expression '<TEST'");
TEST_ERROR(
storagePath(storage, strNew("<TEST>" BOGUS_STR)), AssertError, "'/' should separate expression and path '<TEST>BOGUS'");
// -------------------------------------------------------------------------------------------------------------------------
String *fileExists = strNewFmt("%s/exists", testPath());
TEST_RESULT_INT(system(strPtr(strNewFmt("touch %s", strPtr(fileExists)))), 0, "create exists file");
TEST_RESULT_STR(strPtr(storagePath(storage, strNew("<TEST>"))), "/path/to/test", " expression");
TEST_ERROR(strPtr(storagePath(storage, strNew("<TEST>/"))), AssertError, "path '<TEST>/' should not end in '/'");
TEST_RESULT_BOOL(storageExistsNP(storageTest, fileExists), true, "file exists");
TEST_RESULT_INT(system(strPtr(strNewFmt("sudo rm %s", strPtr(fileExists)))), 0, "remove exists file");
TEST_RESULT_STR(
strPtr(storagePath(storage, strNew("<TEST>/something"))), "/path/to/test/something", " expression with path");
// -------------------------------------------------------------------------------------------------------------------------
if (fork() == 0)
{
sleepMSec(250);
TEST_RESULT_INT(system(strPtr(strNewFmt("touch %s", strPtr(fileExists)))), 0, "create exists file");
exit(0);
}
TEST_ERROR(storagePath(storage, strNew("<NULL>")), AssertError, "evaluated path '<NULL>' cannot be null");
TEST_ERROR(storagePath(storage, strNew("<WHATEVS>")), AssertError, "invalid expression '<WHATEVS>'");
TEST_RESULT_BOOL(storageExistsP(storageTest, fileExists, .timeout = 1), true, "file exists after wait");
TEST_RESULT_INT(system(strPtr(strNewFmt("sudo rm %s", strPtr(fileExists)))), 0, "remove exists file");
}
// *****************************************************************************************************************************
if (testBegin("storageList()"))
{
Storage *storage = NULL;
TEST_ASSIGN(storage, storageNew(strNew(testPath()), 0750, 65536, NULL), "new storage");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
storageList(storage, strNew(BOGUS_STR), NULL, false), PathOpenError,
storageListP(storageTest, strNew(BOGUS_STR), .errorOnMissing = true), PathOpenError,
strPtr(strNewFmt("unable to open directory '%s/BOGUS' for read: [2] No such file or directory", testPath())));
TEST_RESULT_PTR(storageList(storage, strNew(BOGUS_STR), NULL, true), NULL, "ignore missing dir");
TEST_RESULT_PTR(storageListNP(storageTest, strNew(BOGUS_STR)), NULL, "ignore missing dir");
// -------------------------------------------------------------------------------------------------------------------------
String *fileNoPerm = strNewFmt("%s/noperm", testPath());
TEST_RESULT_INT(
system(strPtr(strNewFmt("sudo mkdir %s && sudo chmod 700 %s", strPtr(fileNoPerm), strPtr(fileNoPerm)))), 0,
"create no perm file");
TEST_ERROR(
storageList(storage, fileNoPerm, NULL, false), PathOpenError,
strPtr(strNewFmt("unable to open directory '%s' for read: [13] Permission denied", strPtr(fileNoPerm))));
storageListNP(storageTest, pathNoPerm), PathOpenError,
strPtr(strNewFmt("unable to open directory '%s' for read: [13] Permission denied", strPtr(pathNoPerm))));
// Should still error even when ignore missing
TEST_ERROR(
storageList(storage, fileNoPerm, NULL, true), PathOpenError,
strPtr(strNewFmt("unable to open directory '%s' for read: [13] Permission denied", strPtr(fileNoPerm))));
TEST_RESULT_INT(system(strPtr(strNewFmt("sudo rmdir %s", strPtr(fileNoPerm)))), 0, "create no perm file");
storageListNP(storageTest, pathNoPerm), PathOpenError,
strPtr(strNewFmt("unable to open directory '%s' for read: [13] Permission denied", strPtr(pathNoPerm))));
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(storagePut(storage, strNew("aaa.txt"), bufNewStr(strNew("aaa"))), "write aaa.text");
TEST_RESULT_VOID(
storagePutNP(storageOpenWriteNP(storageTest, strNew("aaa.txt")), bufNewStr(strNew("aaa"))), "write aaa.text");
TEST_RESULT_STR(
strPtr(strLstJoin(storageList(storage, NULL, NULL, false), ", ")), "aaa.txt, stderr.log, stdout.log", "dir list");
strPtr(strLstJoin(storageListNP(storageTest, NULL), ", ")), "aaa.txt, stderr.log, stdout.log, noperm",
"dir list");
TEST_RESULT_VOID(storagePut(storage, strNew("bbb.txt"), bufNewStr(strNew("bbb"))), "write bbb.text");
TEST_RESULT_STR(strPtr(strLstJoin(storageList(storage, NULL, strNew("^bbb"), false), ", ")), "bbb.txt", "dir list");
TEST_RESULT_VOID(
storagePutNP(storageOpenWriteNP(storageTest, strNew("bbb.txt")), bufNewStr(strNew("bbb"))), "write bbb.text");
TEST_RESULT_STR(
strPtr(strLstJoin(storageListP(storageTest, NULL, .expression = strNew("^bbb")), ", ")), "bbb.txt", "dir list");
}
// *****************************************************************************************************************************
if (testBegin("storagePath()"))
{
Storage *storageTest = NULL;
TEST_ASSIGN(storageTest, storageNewNP(strNew("/")), "new storage /");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, NULL)), "/", " root dir");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, strNew("/"))), "/", " same as root dir");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, strNew("subdir"))), "/subdir", " simple subdir");
TEST_ERROR(
storagePathNP(storageTest, strNew("<TEST>")), AssertError, "expression '<TEST>' not valid without callback function");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(
storageTest,
storageNewP(strNew("/path/to"), .pathExpressionFunction = storageTestPathExpression),
"new storage /path/to with expression");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, NULL)), "/path/to", " root dir");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, strNew("/path/to"))), "/path/to", " absolute root dir");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, strNew("is/a/subdir"))), "/path/to/is/a/subdir", " subdir");
TEST_ERROR(
storagePathNP(storageTest, strNew("/bogus")), AssertError, "absolute path '/bogus' is not in base path '/path/to'");
TEST_ERROR(
storagePathNP(storageTest, strNew("/path/toot")), AssertError,
"absolute path '/path/toot' is not in base path '/path/to'");
TEST_ERROR(storagePathNP(storageTest, strNew("<TEST")), AssertError, "end > not found in path expression '<TEST'");
TEST_ERROR(
storagePathNP(storageTest, strNew("<TEST>" BOGUS_STR)), AssertError,
"'/' should separate expression and path '<TEST>BOGUS'");
TEST_RESULT_STR(strPtr(storagePathNP(storageTest, strNew("<TEST>"))), "/path/to/test", " expression");
TEST_ERROR(strPtr(storagePathNP(storageTest, strNew("<TEST>/"))), AssertError, "path '<TEST>/' should not end in '/'");
TEST_RESULT_STR(
strPtr(storagePathNP(storageTest, strNew("<TEST>/something"))), "/path/to/test/something", " expression with path");
TEST_ERROR(storagePathNP(storageTest, strNew("<NULL>")), AssertError, "evaluated path '<NULL>' cannot be null");
TEST_ERROR(storagePathNP(storageTest, strNew("<WHATEVS>")), AssertError, "invalid expression '<WHATEVS>'");
}
// *****************************************************************************************************************************
if (testBegin("storagePathCreate()"))
{
TEST_RESULT_VOID(storagePathCreateNP(storageTest, strNew("sub1")), "create sub1");
TEST_RESULT_INT(storageStatNP(storageTest, strNew("sub1"))->mode, 0750, "check sub1 dir mode");
TEST_RESULT_VOID(storagePathCreateNP(storageTest, strNew("sub1")), "create sub1 again");
TEST_ERROR(
storagePathCreateP(storageTest, strNew("sub1"), .errorOnExists = true), PathCreateError,
strPtr(strNewFmt("unable to create path '%s/sub1': [17] File exists", testPath())));
TEST_RESULT_VOID(storagePathCreateP(storageTest, strNew("sub2"), .mode = 0777), "create sub2 with custom mode");
TEST_RESULT_INT(storageStatNP(storageTest, strNew("sub2"))->mode, 0777, "check sub2 dir mode");
TEST_ERROR(
storagePathCreateP(storageTest, strNew("sub3/sub4"), .noParentCreate = true), PathCreateError,
strPtr(strNewFmt("unable to create path '%s/sub3/sub4': [2] No such file or directory", testPath())));
TEST_RESULT_VOID(storagePathCreateNP(storageTest, strNew("sub3/sub4")), "create sub3/sub4");
TEST_RESULT_INT(system(strPtr(strNewFmt("rm -rf %s/sub*", testPath()))), 0, "remove sub paths");
}
// *****************************************************************************************************************************
if (testBegin("storageOpenRead()"))
{
TEST_ERROR(
storageOpenReadNP(storageTest, strNewFmt("%s/%s", testPath(), BOGUS_STR)),
FileOpenError,
strPtr(strNewFmt("unable to open '%s/%s' for read: [2] No such file or directory", testPath(), BOGUS_STR)));
TEST_RESULT_PTR(
storageOpenReadP(storageTest, strNewFmt("%s/%s", testPath(), BOGUS_STR), .ignoreMissing = true),
NULL, "open missing file without error");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
storageGetNP(storageOpenReadP(storageTest, fileNoPerm, .ignoreMissing = true)), FileOpenError,
strPtr(strNewFmt("unable to open '%s' for read: [13] Permission denied", strPtr(fileNoPerm))));
}
// *****************************************************************************************************************************
if (testBegin("storageOpenWrite()"))
{
TEST_ERROR(
storageOpenWriteNP(storageTest, strNew(testPath())), FileOpenError,
strPtr(strNewFmt("unable to open '%s' for write: [21] Is a directory", testPath())));
// -------------------------------------------------------------------------------------------------------------------------
StorageFile *file = NULL;
String *fileName = strNewFmt("%s/testfile", testPath());
TEST_ASSIGN(file, storageOpenWriteNP(storageTest, fileName), "open file for write (defaults)");
TEST_RESULT_INT(storageStatNP(storageTest, fileName)->mode, 0640, "check dir mode");
close(STORAGE_DATA(file)->handle);
storageRemoveP(storageTest, fileName, .errorOnMissing = true);
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(file, storageOpenWriteP(storageTest, fileName, .mode = 0777), "open file for write (custom)");
TEST_RESULT_INT(storageStatNP(storageTest, fileName)->mode, 0777, "check file mode");
close(STORAGE_DATA(file)->handle);
storageRemoveP(storageTest, fileName, .errorOnMissing = true);
}
// *****************************************************************************************************************************
if (testBegin("storagePut() and storageGet()"))
{
Storage *storageTest = storageNew(strNew("/"), 0750, 65536, NULL);
Storage *storageTest = storageNewP(strNew("/"), .write = true);
TEST_ERROR(
storagePut(storageTest, strNew(testPath()), NULL), FileOpenError,
strPtr(strNewFmt("unable to open '%s' for write: Is a directory", testPath())));
TEST_ERROR(
storageGet(storageTest, strNew(testPath()), false), FileReadError,
storageGetNP(storageOpenReadNP(storageTest, strNew(testPath()))), FileReadError,
strPtr(strNewFmt("unable to read '%s': [21] Is a directory", testPath())));
// -------------------------------------------------------------------------------------------------------------------------
String *fileNoPerm = strNewFmt("%s/noperm", testPath());
TEST_RESULT_INT(
system(strPtr(strNewFmt("sudo touch %s && sudo chmod 600 %s", strPtr(fileNoPerm), strPtr(fileNoPerm)))), 0,
"create no perm file");
TEST_ERROR(
storageGet(storageTest, fileNoPerm, true), FileOpenError,
strPtr(strNewFmt("unable to open '%s' for read: [13] Permission denied", strPtr(fileNoPerm))));
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(storageWriteError(3, 3, strNew("file")), "no write error");
errno = EBUSY;
TEST_ERROR(storageWriteError(-1, 3, strNew("file")), FileWriteError, "unable to write 'file': Device or resource busy");
TEST_ERROR(
storageWriteError(1, 3, strNew("file")), FileWriteError, "only wrote 1 byte(s) to 'file' but 3 byte(s) expected");
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(storagePut(storageTest, strNewFmt("%s/test.empty", testPath()), NULL), "put empty file");
TEST_RESULT_VOID(
storagePutNP(storageOpenWriteNP(storageTest, strNewFmt("%s/test.empty", testPath())), NULL), "put empty file");
// -------------------------------------------------------------------------------------------------------------------------
Buffer *buffer = bufNewStr(strNew("TESTFILE\n"));
StorageFile *file = NULL;
TEST_RESULT_VOID(storagePut(storageTest, strNewFmt("%s/test.txt", testPath()), buffer), "put text file");
StorageFileDataPosix fileData = {.handle = 999999};
MEM_CONTEXT_NEW_BEGIN("FileTest")
{
file = storageFileNew(storageTest, strNew("badfile"), storageFileTypeWrite, &fileData);
}
MEM_CONTEXT_NEW_END();
TEST_ERROR(storagePutNP(file, buffer), FileWriteError, "unable to write 'badfile': [9] Bad file descriptor");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(storageGet(storageTest, strNewFmt("%s/%s", testPath(), BOGUS_STR), false), FileOpenError,
strPtr(strNewFmt("unable to open '%s/%s' for read: [2] No such file or directory", testPath(), BOGUS_STR)));
TEST_RESULT_PTR(storageGet(storageTest, strNewFmt("%s/%s", testPath(), BOGUS_STR), true), NULL, "get missing file");
TEST_RESULT_VOID(
storagePutNP(storageOpenWriteNP(storageTest, strNewFmt("%s/test.txt", testPath())), buffer), "put text file");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(buffer, storageGet(storageTest, strNewFmt("%s/test.empty", testPath()), false), "get empty");
TEST_RESULT_PTR(
storageGetNP(storageOpenReadP(storageTest, strNewFmt("%s/%s", testPath(), BOGUS_STR), .ignoreMissing = true)),
NULL, "get missing file");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(buffer, storageGetNP(storageOpenReadNP(storageTest, strNewFmt("%s/test.empty", testPath()))), "get empty");
TEST_RESULT_INT(bufSize(buffer), 0, "size is 0");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(buffer, storageGet(storageTest, strNewFmt("%s/test.txt", testPath()), false), "get text");
TEST_ASSIGN(buffer, storageGetNP(storageOpenReadNP(storageTest, strNewFmt("%s/test.txt", testPath()))), "get text");
TEST_RESULT_INT(bufSize(buffer), 9, "check size");
TEST_RESULT_BOOL(memcmp(bufPtr(buffer), "TESTFILE\n", bufSize(buffer)) == 0, true, "check content");
@ -171,8 +291,50 @@ testRun()
const Storage *storage = storageTest;
((Storage *)storage)->bufferSize = 2;
TEST_ASSIGN(buffer, storageGet(storageTest, strNewFmt("%s/test.txt", testPath()), false), "get text");
TEST_ASSIGN(buffer, storageGetNP(storageOpenReadNP(storageTest, strNewFmt("%s/test.txt", testPath()))), "get text");
TEST_RESULT_INT(bufSize(buffer), 9, "check size");
TEST_RESULT_BOOL(memcmp(bufPtr(buffer), "TESTFILE\n", bufSize(buffer)) == 0, true, "check content");
}
// *****************************************************************************************************************************
if (testBegin("storageRemove()"))
{
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(storageRemoveNP(storageTest, strNew("missing")), "remove missing file");
TEST_ERROR(
storageRemoveP(storageTest, strNew("missing"), .errorOnMissing = true), FileRemoveError,
strPtr(strNewFmt("unable to remove '%s/missing': [2] No such file or directory", testPath())));
// -------------------------------------------------------------------------------------------------------------------------
String *fileExists = strNewFmt("%s/exists", testPath());
TEST_RESULT_INT(system(strPtr(strNewFmt("touch %s", strPtr(fileExists)))), 0, "create exists file");
TEST_RESULT_VOID(storageRemoveNP(storageTest, fileExists), "remove exists file");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
storageRemoveNP(storageTest, fileNoPerm), FileRemoveError,
strPtr(strNewFmt("unable to remove '%s': [13] Permission denied", strPtr(fileNoPerm))));
}
// *****************************************************************************************************************************
if (testBegin("storageStat()"))
{
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(storagePathCreateP(storageTest, strNew("dir"), .mode = 0777), "create dir with custom mode");
TEST_RESULT_INT(storageStatNP(storageTest, strNew("dir"))->mode, 0777, "check dir mode");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
storageStatNP(storageTest, strNew("missing")), FileOpenError,
strPtr(strNewFmt("unable to stat '%s/missing': [2] No such file or directory", testPath())));
TEST_RESULT_PTR(storageStatP(storageTest, strNew("missing"), .ignoreMissing = true), NULL, "ignore missing file");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
storageStatNP(storageTest, fileNoPerm), FileOpenError,
strPtr(strNewFmt("unable to stat '%s': [13] Permission denied", strPtr(fileNoPerm))));
TEST_RESULT_INT(system(strPtr(strNewFmt("sudo rm -rf %s", strPtr(strPath(fileNoPerm))))), 0, "remove no perm dir");
}
}