* avoid sending mails to addresses which contain the string noreply
* enrich test to ensure that mails containing the string noreply never end up in recipient list
* Update MailSendNotificationTest.groovy
* inject container metadata via env vars to selenium
* set step-specific env var for uiVeri55ExecuteTests
* Revert "set step-specific env var for uiVeri55ExecuteTests"
This reverts commit 8deb39ac5b.
* remove env vars that indicate piper step
* remove env var to indicate execution through selenium step
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* remove defaulting of docker env vars
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Update vars/karmaExecuteTests.groovy
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* writeFile() cannot be passed a Map
I've changed the return type of DebugReport.generateReport() from
String to Map in order to get the generated file name as part of the
return value instead of getting it from a field of DebugReport. The
UnitTest checks whether writeFile() creates the debug_report file
successfully and whether it has the expected contents. The effect
of passing the Map instead of map.contents to writeFile() should
have been an unnecessary wrapping via Map.toString() as in the test,
but in the execution context of Jenkins, this throws an
IllegalArgumentException: Could not instantiate {... and then the
results of map.toString().
* Improve JenkinsWriteFileRule compatibility
Calling m.text.toString() is wrong, since the type stored at m.text
already needs to be a String (or GString). Expecting valid parameters
here makes sure problems are detected by tests already. (All tests
pass as before.)
* Protecode as go implementation
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
The code of extensions was not executed within the try-catch-block of
handlePipelineStepErrors. The main benefit of this change is better
logging and re-using the 'unstable' feature also for
extended/overwritten steps.
This should have been added along with DebugReport. For context:
EnvironmentUtils used to be a class alongside DebugReport and thus
there was no 'import' directive. Working with Jenkins DSL stuff
has numbed my ability to pay attention to the IDE indicating errors.
The DebugReport is a global instance where steps can store information relevant for diagnosing failed pipelines. In the SDK Pipeline, this is used to generate a debug report within the postActionArchiveDebugLog step. The reason for adding this to Piper is to feed information about extended or overwritten stages in piperStageWrapper into the DebugReport, as was done before in the SDK Pipeline's equivalent runAsStage step.
* add evaluateFromMavenPom to piper Utils
* adapt mavenExecute to accept `returnStdout` as parameter. If configured mavenExecute will return the stdout for further processing
* adapt tests of mavenExecute and mavenArtifactVersioning as well as add another exception to CommonStepsTest because mavenExecute will return a String if configured
* Fix random UUID generation in Unit Tests
* Use before and after methods
* Add import of org.junit.After
* Add go step for abapEnvironmentPullGitRepo
* Revert "Add go step for abapEnvironmentPullGitRepo"
This reverts commit 08b6b16217.
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
On groovy level we take into account:
- parameters via signature
- project config
- landscape config(s)
- for docker: nested and plain config
also more tests added ...
* Use commonPipelineEnvironment in go binary
* Update groovy part incl. tests
* Rework structure and naming
* Support influx resources in steps
* Update tests and some cleanups
* Add correct defer handling
* Address PR feedback
* Fix test
* Update resources.go
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Add new config mtarName for mtaBuild step
* Remove unnecessary whitespace changes in unit test
* Sort new config & avoid file operation when this config provided
* Modify the test to take the custom name without extension
* Update new config documentation
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* custom mta name should be given with mtar extension
* Updated the config documentation
* Add PiperGoUtils for downloading piper binary
PiperGoUtils provide the link between a Jenkins library step and the library step execution running in a go binary.
It makes sure that an adequate binary is available.
* fix CodeClimate finding
* Remove Delimiter and add download resilience.
* tests(testsPublishResults): evaluate file pattern for cobertura
* tests(testsPublishResults): test for cobertura in UI5 projects
* feat(testsPublishResults): collect cobertura of UI5 projects by default
* [refactoring] condence common coding for cf deploy
Small change beyond refactoring: for mtaDeploy the user is now quoted.
* more general name: logoutAction -> postDeployAction
# Changes
This PR adds a new step: cloudFoundryServiceCreate
There is a cf community plugin [Create-Service-Push](https://github.com/dawu415/CF-CLI-Create-Service-Push-Plugin) available to apply infrastructure as code to Cloud Foundry. The plugin uses a manifest.yml to create services in a targeted CF space.
The proposed step provides an interface to this plugin.
Already done:
- [x] Tests
- [x] Documentation
Further actions:
- a Refactoring: Move varOptions and varsFileOption code into a class and make us of this here and in cloudFoundryDeploy step. -> Is it ok to use the CfManifestUtils, or add it as a new class to variablesubstitiion package?
- enhance the s4sdk cf cli docker image to include the plugin.
* Provide additional opts for cf deploy
Inside cloudFoundyDeploy we use these cf commands
o login
o plugins
o blue-green-deploy
o push
o deploy
o bg-deploy
o stop
o logout
o logout and stop does not provide any options
o plugins provides options (--checksum --outdated) but it is unlikely that
these options can be used in a reasonable way during the deploy process.
o login now uses `loginOpts`
o The other commands uses now `deployOpts`
* provide additional opts also for cf api calls
* Provide more log when verbose
* re-use mtaDeployParameters and adjust names of other params (api, login) accordingly
* Streamline naming
* distinuish between cfNative and mta deploy params
* Add cfNativeDeployParam default
* login and api paramters are not under cloudFoundry
* Back commonPipelineEnvironment step by shared class
Each pipeline step comes with its own instance of a commonPipelineEnvironment.
Properties stored on one instance was not shared with the other instances.
Now we strip down the commonPipelineEnvironment step and forward basically
everything to a shared singleton instance.
With that approach all instances of commonPipelineEnvironment shares the
same data and can now be really used for information exchange between the steps.
Before that change only the commonPipelineEnvironment instance associated with
the pipeline script itself could be used for that purpose.
* Remove unneeded commented line
* Changes:
- New YamlSubstituteVariables step to substitute variables in YAML files with values from another YAML
- New Tests, that check the different substitution patterns.
- Added test resources, including various manifest and variables files.
- Improved usage of JenkinsLoggingRule
- Improved JenkinsReadYamlRule to properly reflect the mocked library's behaviour.
- Added a new JenkinsWriteYamlRule.
* Changes:
- added a Logger that checks a config.verbose flag before it logs debug messages.
- changed error handling to rethrow Yaml parsing exception in case of wrongly-formatted Yaml files.
- changed JenkinsWriteYamlRule to capture Yaml file details of every invocation of writeYaml. This allows sanity checks at end of tests, even if there were multiple invocations.
- adjusted tests.
* Changes:
- Removed javadoc-code blocks from API documentation since they are not supported.
- Removed skipDeletion boolean.
- Added a new deleteFile script which deletes a file if present.
- Added a new JenkinsDeleteFileRule to mock deleteFile script and optionally skip deletion for tests.
- Adjusted yamlSubstituteVariables script.
- Adjusted tests to include new JenkinsDeleteFileRule.
- Changed code that deletes an already existing output file to produce better logs.
* Changes:
- Turned yamlSubstituteVariables into a script that works purely based on Yaml data (not files).
- Added a new cfManifestSubstituteVariables that uses yamlSubstituteVariables under the hood but works based on files.
- Adjusted tests, and added new ones.
* Adjusted documentation and a few log statements.
* Changed documentation to no longer include javadoc code statements.
* Made mocking of deletion of a file a default. Adjusted tests.
* Changed signature of yamlSubstituteVariables' call method to return void.
* Changes:
- Fixed naming issues in deleteFile.
- Renamed Logger to DebugHelper.
- Fixed some documentation.
* Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.
* PROPERLY Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.
* Changes:
- Added tests for deleteFile script
- Changed JenkinsFileExistsRule to also keep track of which files have been queried for existence.
* Changes:
- Removed java.io.File usage from cfManifestSubstituteVariables and using fileExists instead now.
- Adjusted tests.
* Wrapped file path inside ticks to allow spaces in file path when calling deleteFile.
* Removed null checks of mandatory parameters, and resorted to ConfigurationHelper.withMandatoryProperty
* Fixed a NullPointer due to weird Jenkins / Groovy behaviour.
* Changes:
- Turned yamlSubstituteVariables step into a utils class.
- Added tests
- Adjusted cfManifestSubstituteVariables to use utils class instead of step.
- Adjusted tests
- Adjusted APIs of DebugHelper.
* Re-introduced log statement that shows what variables are being replaced and with what.
* Changing API of YamlUtils to take the script and config as input.
* Test
* Test
* Test
* Test
* Test
* Fixing issue.
* Fixing issue.
* Changes:
- Refactored DebugHelper and YamlUtils to make usage nicer and rely on dependency injection.
- Removed Field for DebugHelper and turned it into local variable.
- Adjusted classes using the above.
- Adjusted tests where necessary.
* Added link to CF standards to YamlUtils also.
* Add docu for step cfManifestSubstituteVariables.md
* Added documentation.
* Added missing script parameter to documentation. Some steps document it, some don't. Right now you need it, so we document it.
* Fixed some layouting and typos
* Beautified exception listing.
* Removed trailing whitespaces to make code climate checks pass.
* Trying to get documentation generated, with all the exceptions to markup one should not use.
* cosmetics.
* cosmetics, part 2
* Code climate changes...
* Inlined deleteFile step.
* Added two more tests to properly check file deletion and output handling.
* Changes:
- adjusted API to take a list of variables files, as does 'cf push --vars-file'
- adjusted API to allow for an optional list of variable key-value-maps as does 'cf push --vars'
- reproduced conflict resolution and overriding behavior of variables files and vars lists
- adjusted tests and documentation
* Added missing paramter to doc comment.
* Re-checked docs for missing paramters or params that have no counterpart in the method signature.
* Adjusted documentation.
* Removed absolute path usage from documentation.
* corrected documentation.
* Changed javadoc comment to plain comment.
* Turned all comments to plain comments.
Before: complete scmInfo was handed over via method signature.
After: Only the relevant part (GIT_URL from scmInfo) is handed over.
All the other properties from scmInfo are not used in the method body.
With this appraoch it is more obvious what is used inside the method.
* dockerExecuteOnKubernetes - add stashBack configuration
For certain cases it is valuable to only bring back some of the files from an execution inside a container back to the workspace.
This is now added.
Closes#753
* refactor according to PR review
Each pipeline step comes with its own instance of a commonPipelineEnvironment.
Properties stored on one instance was not shared with the other instances.
Now we strip down the commonPipelineEnvironment step and forward basically
everything to a shared singleton instance.
With that approach all instances of commonPipelineEnvironment shares the
same data and can now be really used for information exchange between the steps.
Before that change only the commonPipelineEnvironment instance associated with
the pipeline script itself could be used for that purpose.
* Ensure closure gets called when neither returnStdout nor returnStatus are set
In this case we do not have a return value, but in case we execute a closure we should execute the closure.
With that it is possible to raise an exception from the closure.
* [refactoring] unify usage of unify method call
* Remove dead code.
Coding after uncondition throw exception statement does not get executed.
* Ensure script rule behaves the same whan called with string and with map.
* Take proper jnlp image as default for Kubernetes execution
Following changes are contained:
* removal of custom jnlp image as default
* allow customization of jnlp image via system environment
fixes#757
* add documentation
This step should serve as generic entry point in pipelines for building artifacts.
Build principle: build once.
Purpose of the step:
- build using a defined build technology
- store build result for future use in testing etc.
* dockerExecuteOnKubernetes - hide yaml by default
* hide step parameters to not leak sensitive parameter values into the log
* add more details to log output
Fix issue when retrieving version via tag in `FROM` section.
We also need to consider that the registry may contain a port.
* Update DockerArtifactVersioning.groovy
* add error message for missing image tag in FROM line
stashing .git directory had negative side-effects.
Solution would be to stash `.git` folder and unstash in `dockerExecuteOnKubernetes` only if required for a dedicated scenario.
The config map prepared by ConfigHelper is a mix from several configuration levels. The lowest config level
(DefaultValueCache) is shared between several ConfigHelper invocations. In case a Map or Collection which is
inherited from the DefaultValueCache level gets modified, this is also visible for all subsequent steps. This
causes trouble and situation which are hard to debug.
With this change here each invocation of ConfigHelper.use() provides a deep defensive copy. With that we can
ensure that there is no configuration update from one step to another.
* add Slack notification to post stage
* add Slack notification to init stage
* add trigger condition for Slack notification
* fix whitespaces
* use capital stage name
* add tests for init stage
* remove unused import
* add tests for post stage
* minor changes
* fix typo
* whitesourceExecuteScan - fix stashing behavior for golang
**changes:**
golang scan requires sources to be present. They are already part of the `checkmarx` stash which can be used here.
* fix tests
extends init condition with condition `configKeys`
This condition allows to specify a list of configuration keys which if any key is set will activate the respective step & stage
Allow setting custom settings file for maven in mta build, which is for example required if a custom maven repo (i.e. company internal) needs to be used.
* add new step buildSetResult
* set pipeline result in post stage
* exclude buildSetResult from commonStepTests
* extend pipeline test
* remove post stage reference
In the meantime we check earlier if the deployable exists. Up to now the test used that late check. Now, as that
check happens earlier in the code we have to tune here. Now we assume a failure inside neo deploy instead.
* handlePipelineStepErrors - allow step timeouts
This adds another resilience option:
A timeout can be configured for steps in order to stop step execution, continue with the pipeline while setting build status to "UNSTABLE"
* alpine does not support date option --universal
Replaced by --utc as this seems to be more universal than --universal
* Fix unit tests after date parameter change
Before the test we remmber which test is currently running.
During the test we collect all the calls to steps.
Beside that we persist the names of all steps within this shared
lib itself.
After the test(s) we write a corresponding json file. In fact we
write the file after each test, which is too often. But since we
don't know which test is the last test we can't do better.
The resulting file can be used later on for resolving the plugins
contributing the particular steps.
With that we are able to create a list of required plugins for each
step.
* Fix sanity checks for warPropertiesFile deploy mode.
* improve tests for the sanity checks
The sanity checks are performed per deploy mode.
All parameters are checked at once.
* Explict check for host, account not found by sanity checks for deploy mode war properties
* Define pod using k8s yaml manifest
The Kubernetes plugin allows to define pods directly via the Kubernetes
API specification:
https://github.com/jenkinsci/kubernetes-plugin#using-yaml-to-define-pod-templates
This has the advantage of unlocking Kubernetes features which are not
exposed via the Kubernetes plugin, including all Kubernetes security
featues.
Using the Kubernetes API directly is also better from development
point of view because it is stable and better desgined then the API the
plugin offers.
* Make the Kubernetes ns configurable
If one Jenkins Master is used by multiple Teams, it is desirable to
schedule K8S workloads in seperatae workspaces.
* Add securityContext to define uid and fsGroup
In the context of the Jenkins k8s plugin it is uids and fsGroups play an
important role, because the containers share a common file system.
Therefore it is benefical to configure this data centraly.
* fix indention
* Undo format changes
* Extend and fix unit tests
* Fix port mapping
* Don't set uid globally
This does not work with jaas due to permissions problems.
* Fix sidecar test
* Make security context configurable at stage level
* Extract json serialization
* Cleanup unit tests
In case of a mis-configuration we get a hint like "host is missing".
Actually it should be "neo/host is missing" since the parameter "host" is nested inside "neo".
Having simply "host" confuses the person troubleshooting this issue.
With this change the input validation is performed right at the beginning of the step.
The NeoCommandLine helper does not check a second time now.
In case unstashAll is called with a Set containing a null value we get
an ugly message in the log:
Unstash failed: null (Could not instantiate {name=null} for UnstashStep(name: String):
This can be avoided by unstashing only in case we have an ID.
options are not supported for dockerExecuteOnKubernetes, but
environment variables are. Hence we switch to environment variables
which works on both, plain docker and on kubernetes.
Using env vars is the cleaner approach anyway.
Nobody will have a look at println statements emitted by tests. Proper way is to assert.
* log is empty, not sure if it makes sense to assert that the log is empty.
* result is already asserted, so everything is fine there ...
We should try to avoid promting statements to the log during tests. This is basically
some kind of manual/visual checking of conditions. This does not work since in most cases
nobody scans test logs manually/visually. I think this is often a smell for missing assertions.
* Provide flag for skipping docker image pulls
Usefull for testing while developing images. Without pulling images it is possible to
use images available in the local registry only.
* Project "Piper" Template: framework and structure
This is the first in a series of Pull-Requests to provide a fully functional Pipeline template into the library.
This PR contains:
* general Pipeline structure incl. tests
* Pipeline initialization incl. tests
* Hollow pipeline stages incl. tests
* first set of default values
* add gitUrls to commonPipelineEnvironment
* set correct git information on commonPipelineEnvironment
