The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".
To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.
We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.
Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.
Also fix an outdated comment.
This function was included in a header but not declared inline, so linker errors happened when the header was included into more than one file.
Because of the setjmp() in TRY_BEGIN() it can't be inlined so put it in a C file.
Also add some missing headers.
There have been intermittent failures on f33 (with coverage) but not on u16 (without coverage).
Reproducing this reliably has been very difficult, so just try increasing the timeouts. This is based on the observation that tests with coverage take longer than tests without, which may lead the f33 tests to fail if CI is running slower than usual.
This will not increase the runtime of the test unless there is an error.
If configure/make has been run in the src path it can conflict with tests, which may require different build options.
Also add a comment when rebuilding for code generation.
This file duplicated the command list that already exists in parse.auto.c.
Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.
Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.
This was started in c5ae047e but did not include generation of parse.auto.c.
The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.
Option order resolution now includes cycle detection.
Remove strIdGenerate() since bldStrId() performs the same function without cluttering the core code. Since bldStrId() is intended to work in non-debug builds, move the validity checks for input strings out of the DEBUG block.
StringIds are generated as 5/6 bit, whichever is most efficient, for each option value. cfgOptionStrIdInternal() has been updated for this logic.
This allows a local/remote to be started independently of server initialization, which will be useful for implementing new transport types, e.g. TLS.
Also remove some dead code in localTest.c.
protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()).
The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.
Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.
This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c
--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.
Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.
This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.
The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.
Add --no-install-recommends to apt-get commands to save a bit of time and space.
Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.
Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.
Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.
Includes backup and backupCommon tests.
Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.
String values were also removed where they were no longer needed.
It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.
Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.
This removes a lot of boiler plate where every instance needs to create these interfaces.
Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes.
In both cases update the tests with the new macros.
Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.
Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
The protocol does not put an end message on exit so there was a race between the main process exiting and the remote processes exiting. If the main process exited first then the remote processes might not write coverage data causing coverage to fail.
Fix by calling exit explicitly at the end of the test and update the harness to put an end message so the exits are synchronized.
In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:
1. to ensure that an inadvertent typo in the path/file name does not go undetected,
2. to ensure that nothing else has removed the file prior to the call, and
3. consistency
Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.
