This change adds the capabilities to check if the user still uses legacy
configuration parameters and allows to fail the pipeline in the
piperPipelineStageInit in case incompatible configuration is found.
Thus, it is now possible to describe breaking changes in the
configuration with the use of a yaml resource file. In this file, one can
describe the changes in configuration, e.g., when a configuration parameter
was replaced or removed. A concrete example for such a file for the Cloud
SDK Pipeline is part of this PR.
This change add the support for running integration tests using the
recently introduced step mavenExecuteIntegration in
piperPipelineStageIntegration. In addition, capabilities to provide
temporary credentials to these tests is added using the
writeTemporaryCredentials step.
This change adds a utils class to write credentials specified in Jenkins to a
temporary file, to be used for the execution of, e.g., integration tests
as already available in Cloud SDK Pipeline.
* add APIs for getNodes, getExtDescriptor, putExtDescriptor and uploadExtDescriptor
* add new params and extend the logic to enable upload or update extension descriptor to node
* extends unit tests
Co-authored-by: Zihe Cheng <zihe.cheng@sap.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
* Fix docker.includes in WhitesourceConfigurationHelper
Not sure, but it seems the [`docker.includes` parameter](https://whitesource.atlassian.net/wiki/spaces/WD/pages/804814917/Unified+Agent+Configuration+File+and+Parameters#UnifiedAgentConfigurationFileandParameters-DockerImages) needs to be a regex.
Our pipeline is failing with:
```
10:22:33 [ERROR] [2020-06-15 08:22:33,740 +0000] - Resolve DockerEntity Exception Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
10:22:33 [DEBUG] [2020-06-15 08:22:33,743 +0000] - Resolve DockerEntity Exception {}
10:22:33 java.util.regex.PatternSyntaxException: Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
10:22:33 at java.base/java.util.regex.Pattern.error(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.sequence(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.expr(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.compile(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.<init>(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.compile(Unknown Source)
10:22:33 at org.whitesource.utils.WssStringUtils.isMatchingPattern(WssStringUtils.java:49)
10:22:33 at org.whitesource.agent.dependency.resolver.docker.DockerResolver.filterTarImagesToScan(DockerResolver.java:296)
10:22:33 at org.whitesource.agent.dependency.resolver.docker.DockerResolver.resolveDockerEntities(DockerResolver.java:186)
10:22:33 at org.whitesource.fs.scanOrigins.DockerEntityScanOrigin.scan(DockerEntityScanOrigin.java:66)
10:22:33 at org.whitesource.fs.scanOrigins.ScanOrigin.runOriginScan(ScanOrigin.java:36)
10:22:33 at org.whitesource.fs.FileSystemAgent.createProjects(FileSystemAgent.java:132)
10:22:33 at org.whitesource.fs.Main.scanAndSend(Main.java:157)
10:22:33 at org.whitesource.fs.Main.main(Main.java:102)
10:22:33 [WARN] [2020-06-15 08:22:33,744 +0000] - Resolve DockerEntity Exception Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
```
* Switch docker.includes to slashy string
* Fix docker includes pattern in tests
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: D070410 <srinikitha.kondreddy@sap.com>
* fetch xsuaa credentials in newman execution
* refactor CfUtils and moving to integration package
* removing CfUtils from old package com.sap.piper
* avoid config.verbose as "null"
* variable renamed for consistent naming
* handling NPE and other comments incorp
* added test cases
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Provide test for yamlUtils substitution with boolean value false
* Fix: handle booleans with value 'false' in yaml substitution
* Improve variable naming
* write flag for sonar execution to influx
* add missing step metadata file
* first attempt to read influx from disk
* add missing import
* correct glob pattern
* use file path
* correct type names
* cleanup
* fix code climate issue
* fix typo
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Avoid maven error `Unknown lifecycle phase \"-\"` when the value of a define contains `-`.
Don't split and trim maven arguments. Expect they come in as a list, keep them as list.
This is a breaking change compared to the old Groovy implementation which relied on using a shell for calling maven.
As an example, consider this diff:
```diff
- goals: 'org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate',
- defines: "-Dexpression=$pomPathExpression -DforceStdout -q",
+ goals: ['org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate'],
+ defines: ["-Dexpression=$pomPathExpression", "-DforceStdout", "-q"],
```
Also establish mavenExecute() via new JenkinsMavenExecuteRule in Groovy Unit Tests.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Extend JenkinsLoggingRule and TransportManagementServiceTest
* Print response content for status codes >= 300, also test that it's not
revealed in the case of unexpected status codes between 200 and 300
(both not included) for authentication
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
* Provide response from tms file upload also in case of return codes outside 2xx
* Update src/com/sap/piper/integration/TransportManagementService.groovy
Co-Authored-By: Oliver Feldmann <oliver.feldmann@sap.com>
* Update src/com/sap/piper/integration/TransportManagementService.groovy
Co-Authored-By: Oliver Feldmann <oliver.feldmann@sap.com>
* consider re-running in verbose mode only if we are not in verbose mode
* Add missing script reference when calling error
* avoid curl --fail in order to get also a response in case of 4xx 5xx
* add missing write-out
* --output instead of -o
* fix syntax errors
* fix codeclimat issue
* fix unit tests /1
* Adjust unit tests
* More unit tests
* Use other texts in verbose mode and non verbose mode in case of a failure
in order to avoid issue with hanging log messages surviving a test case (... should not be the case).
For the non verbose mode we check the http response code since there is not message we can check.
* Now with the full comment explaining the 418
* Provide different responses for verbose and non-verbose mode
in order to distinguish the cases
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
* Update schema and hcp deployer version in mta.yaml file
* Add name parameter in mta yaml file
* Rename template_mta.yml to template_mta.yaml
* Fix indentation
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
* Protecode as go implementation
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
This should have been added along with DebugReport. For context:
EnvironmentUtils used to be a class alongside DebugReport and thus
there was no 'import' directive. Working with Jenkins DSL stuff
has numbed my ability to pay attention to the IDE indicating errors.
The DebugReport is a global instance where steps can store information relevant for diagnosing failed pipelines. In the SDK Pipeline, this is used to generate a debug report within the postActionArchiveDebugLog step. The reason for adding this to Piper is to feed information about extended or overwritten stages in piperStageWrapper into the DebugReport, as was done before in the SDK Pipeline's equivalent runAsStage step.
* add evaluateFromMavenPom to piper Utils
* adapt mavenExecute to accept `returnStdout` as parameter. If configured mavenExecute will return the stdout for further processing
* adapt tests of mavenExecute and mavenArtifactVersioning as well as add another exception to CommonStepsTest because mavenExecute will return a String if configured
* Add PiperGoUtils for downloading piper binary
PiperGoUtils provide the link between a Jenkins library step and the library step execution running in a go binary.
It makes sure that an adequate binary is available.
* fix CodeClimate finding
* Remove Delimiter and add download resilience.
* Changes:
- New YamlSubstituteVariables step to substitute variables in YAML files with values from another YAML
- New Tests, that check the different substitution patterns.
- Added test resources, including various manifest and variables files.
- Improved usage of JenkinsLoggingRule
- Improved JenkinsReadYamlRule to properly reflect the mocked library's behaviour.
- Added a new JenkinsWriteYamlRule.
* Changes:
- added a Logger that checks a config.verbose flag before it logs debug messages.
- changed error handling to rethrow Yaml parsing exception in case of wrongly-formatted Yaml files.
- changed JenkinsWriteYamlRule to capture Yaml file details of every invocation of writeYaml. This allows sanity checks at end of tests, even if there were multiple invocations.
- adjusted tests.
* Changes:
- Removed javadoc-code blocks from API documentation since they are not supported.
- Removed skipDeletion boolean.
- Added a new deleteFile script which deletes a file if present.
- Added a new JenkinsDeleteFileRule to mock deleteFile script and optionally skip deletion for tests.
- Adjusted yamlSubstituteVariables script.
- Adjusted tests to include new JenkinsDeleteFileRule.
- Changed code that deletes an already existing output file to produce better logs.
* Changes:
- Turned yamlSubstituteVariables into a script that works purely based on Yaml data (not files).
- Added a new cfManifestSubstituteVariables that uses yamlSubstituteVariables under the hood but works based on files.
- Adjusted tests, and added new ones.
* Adjusted documentation and a few log statements.
* Changed documentation to no longer include javadoc code statements.
* Made mocking of deletion of a file a default. Adjusted tests.
* Changed signature of yamlSubstituteVariables' call method to return void.
* Changes:
- Fixed naming issues in deleteFile.
- Renamed Logger to DebugHelper.
- Fixed some documentation.
* Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.
* PROPERLY Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.
* Changes:
- Added tests for deleteFile script
- Changed JenkinsFileExistsRule to also keep track of which files have been queried for existence.
* Changes:
- Removed java.io.File usage from cfManifestSubstituteVariables and using fileExists instead now.
- Adjusted tests.
* Wrapped file path inside ticks to allow spaces in file path when calling deleteFile.
* Removed null checks of mandatory parameters, and resorted to ConfigurationHelper.withMandatoryProperty
* Fixed a NullPointer due to weird Jenkins / Groovy behaviour.
* Changes:
- Turned yamlSubstituteVariables step into a utils class.
- Added tests
- Adjusted cfManifestSubstituteVariables to use utils class instead of step.
- Adjusted tests
- Adjusted APIs of DebugHelper.
* Re-introduced log statement that shows what variables are being replaced and with what.
* Changing API of YamlUtils to take the script and config as input.
* Test
* Test
* Test
* Test
* Test
* Fixing issue.
* Fixing issue.
* Changes:
- Refactored DebugHelper and YamlUtils to make usage nicer and rely on dependency injection.
- Removed Field for DebugHelper and turned it into local variable.
- Adjusted classes using the above.
- Adjusted tests where necessary.
* Added link to CF standards to YamlUtils also.
* Add docu for step cfManifestSubstituteVariables.md
* Added documentation.
* Added missing script parameter to documentation. Some steps document it, some don't. Right now you need it, so we document it.
* Fixed some layouting and typos
* Beautified exception listing.
* Removed trailing whitespaces to make code climate checks pass.
* Trying to get documentation generated, with all the exceptions to markup one should not use.
* cosmetics.
* cosmetics, part 2
* Code climate changes...
* Inlined deleteFile step.
* Added two more tests to properly check file deletion and output handling.
* Changes:
- adjusted API to take a list of variables files, as does 'cf push --vars-file'
- adjusted API to allow for an optional list of variable key-value-maps as does 'cf push --vars'
- reproduced conflict resolution and overriding behavior of variables files and vars lists
- adjusted tests and documentation
* Added missing paramter to doc comment.
* Re-checked docs for missing paramters or params that have no counterpart in the method signature.
* Adjusted documentation.
* Removed absolute path usage from documentation.
* corrected documentation.
* Changed javadoc comment to plain comment.
* Turned all comments to plain comments.
