This change adds the capabilities to check if the user still uses legacy
configuration parameters and allows to fail the pipeline in the
piperPipelineStageInit in case incompatible configuration is found.
Thus, it is now possible to describe breaking changes in the
configuration with the use of a yaml resource file. In this file, one can
describe the changes in configuration, e.g., when a configuration parameter
was replaced or removed. A concrete example for such a file for the Cloud
SDK Pipeline is part of this PR.
This change adds the support for executing maven static code checks and
linting using the mavenExecuteStaticCodeChecks and npmExecuteLint steps
as part of the piperPipelineStageBuild.
Co-authored-by: Florian Wilhelm <florian.wilhelm02@sap.com>
This change extends the testsPublishResults step to support publishing
of HTML reports using the HTML publisher plugin. This change is a
preperation for the migration of frontend unit tests as already existing
in the SAP Cloud SDK Pipeline to the
piperPipelineStageAdditionalUnitTests.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
This change add the support for running integration tests using the
recently introduced step mavenExecuteIntegration in
piperPipelineStageIntegration. In addition, capabilities to provide
temporary credentials to these tests is added using the
writeTemporaryCredentials step.
This change adds the groovy step writeTemporaryCredentials, in order to
avoid resolving configuration parameters within the piperPipelineStageIntegration
in the future.
For the integration tests functionality provided by the SAP Cloud SDK
Pipeline it needs be possible to configure a credential id pointing to
Jenkins credentials, which are then temporarily written to a file to be
available during the execution of the integration tests.
With the previous implementation solely in the TemporaryCredentialsUtils
class, it would be required to resolve the credentialId from the
configuratuion within the piperPipelineStageIntegration, once we fully
migrate the Cloud SDK integration tests stage.
This change adds a utils class to write credentials specified in Jenkins to a
temporary file, to be used for the execution of, e.g., integration tests
as already available in Cloud SDK Pipeline.
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Adapt changes of abautils pkg
* Add test for polling
* Minor fix
* Fix yaml spacing
* Add longdescription to yaml
* Refactor abaputil methods
* Refactoring
* Refactoring
* Minor fix
* Minor fixeds
* Adapt to new abaputils.AUtilsMock
* Delete obsolete initial checks for params
* Fix manageGitRepoUtils_test.go
* Adjust pollEntity tests
* Add abapEnvironmentCheckoutBranch.groovy file
* Add documentation for abapEnvCheckoutBranch step
* Add step to whitelist in commonStepsTest.groovy
* Add GENERAL scope to host parameter
* Fix generated.go file
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
This change enables the use of the new npmExecuteEndToEndTests in the
stages release and acceptance if it is configured. If it is not
configured, nothing should change.
This change adds a buildDescriptorExcludeList parameter to
npmExecuteScripts, to enable the exclusion of certain directories when
executing npm scripts. Previously, npmExecuteScripts could only execute
scripts in all packages.
Now it is possible to provide paths or patterns as elements of the
buildDescriptorExcludeList to exclude packages when executing npm scripts.
This change adds the missing report handling to npmExecuteEndToEndTests
step. In addition, it extends the testsPublishResults step to support
cucumber test results.
This change adds the end to end test execution capabilities of the SAP
Cloud SDK pipeline, i.e., the consumer can define a npm script which is
then executed using the npmExecuteScripts step.
The handling of generated results will be provided in a follow up PR.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
The multicloudDeploy step is newer and more capable compared to the old steps. For pipelines that don't configure the multicloudDeploy step, nothing should change.
Previously, all logs were stored in the same folder and in case of an
error all logs in that folder would be returned to the user. Thus, it
is possible that unrelated logs, i.e., from different deployments (in
case of parallel deployments) are returned to the user.
* add APIs for getNodes, getExtDescriptor, putExtDescriptor and uploadExtDescriptor
* add new params and extend the logic to enable upload or update extension descriptor to node
* extends unit tests
Co-authored-by: Zihe Cheng <zihe.cheng@sap.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
This change adds linting support for Java-/TypeScript projects based on the checkDefaultLint and checkUserLint steps from Cloud SDK Pipeline.
Following options are supported:
1. Define a linting script named ci-lint in the package.json file(s) of the project
2. Provide a configuration for ESLint as part of the project
3. If none of the above: ESLint is executed with a general purpose configuration.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Fix docker.includes in WhitesourceConfigurationHelper
Not sure, but it seems the [`docker.includes` parameter](https://whitesource.atlassian.net/wiki/spaces/WD/pages/804814917/Unified+Agent+Configuration+File+and+Parameters#UnifiedAgentConfigurationFileandParameters-DockerImages) needs to be a regex.
Our pipeline is failing with:
```
10:22:33 [ERROR] [2020-06-15 08:22:33,740 +0000] - Resolve DockerEntity Exception Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
10:22:33 [DEBUG] [2020-06-15 08:22:33,743 +0000] - Resolve DockerEntity Exception {}
10:22:33 java.util.regex.PatternSyntaxException: Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
10:22:33 at java.base/java.util.regex.Pattern.error(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.sequence(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.expr(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.compile(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.<init>(Unknown Source)
10:22:33 at java.base/java.util.regex.Pattern.compile(Unknown Source)
10:22:33 at org.whitesource.utils.WssStringUtils.isMatchingPattern(WssStringUtils.java:49)
10:22:33 at org.whitesource.agent.dependency.resolver.docker.DockerResolver.filterTarImagesToScan(DockerResolver.java:296)
10:22:33 at org.whitesource.agent.dependency.resolver.docker.DockerResolver.resolveDockerEntities(DockerResolver.java:186)
10:22:33 at org.whitesource.fs.scanOrigins.DockerEntityScanOrigin.scan(DockerEntityScanOrigin.java:66)
10:22:33 at org.whitesource.fs.scanOrigins.ScanOrigin.runOriginScan(ScanOrigin.java:36)
10:22:33 at org.whitesource.fs.FileSystemAgent.createProjects(FileSystemAgent.java:132)
10:22:33 at org.whitesource.fs.Main.scanAndSend(Main.java:157)
10:22:33 at org.whitesource.fs.Main.main(Main.java:102)
10:22:33 [WARN] [2020-06-15 08:22:33,744 +0000] - Resolve DockerEntity Exception Dangling meta character '*' near index 0
10:22:33 *.tar
10:22:33 ^
```
* Switch docker.includes to slashy string
* Fix docker includes pattern in tests
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: D070410 <srinikitha.kondreddy@sap.com>
* fetch xsuaa credentials in newman execution
* refactor CfUtils and moving to integration package
* removing CfUtils from old package com.sap.piper
* avoid config.verbose as "null"
* variable renamed for consistent naming
* handling NPE and other comments incorp
* added test cases
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Provide test for yamlUtils substitution with boolean value false
* Fix: handle booleans with value 'false' in yaml substitution
* Improve variable naming
* piperExecuteBin - prevent non-speaking exit code 1 error
Make sure that error during context resolution gets more context.
* properly provide golang errors
* improve error message, when no details are available
* Add error category
* Update getConfig.go
* streamlined error category with standard term
* remove unused import from test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* added new step gctsDeployCommit
* suggested PR fixes applied
* fixed test
* Remove unused imports
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* added URL encoding for 'request' parameter
* regenerate after change
* minor changes
* renamed step to 'gctsDeploy'
* changed name
* remove space
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* write flag for sonar execution to influx
* add missing step metadata file
* first attempt to read influx from disk
* add missing import
* correct glob pattern
* use file path
* correct type names
* cleanup
* fix code climate issue
* fix typo
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Add parameter "--ignoreCustomDefaults"
* Pass to piper customDefaults from config also via --defaultConfig
... and add "--ignoreCustomDefaults".
* Log output when ignoring customDefaults
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
This change enables the setupCommonPipelineEnvironment step to handle
custom default configurations defined in customDefaults parameter of the
project configuration.
Previously, only the getConfig Go step was able to incorporate custom
default configurations.
Update documentation on custom defaults and sharing between projects.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Change Groovy implementation to piperExecuteBin()
This commit changes the old groovy implementation to the new way using the piperExecuteBin() function.
* Added abapEnvironemtPullGitRepo to whitelist for Tests
* Changed groovy implementation to using piperExecuteBin()
* Added cfDeleteService to test whitelist
* Removed Groovy unit tests
* Remove whitelist tests
* Update test/groovy/CommonStepsTest.groovy
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Avoid maven error `Unknown lifecycle phase \"-\"` when the value of a define contains `-`.
Don't split and trim maven arguments. Expect they come in as a list, keep them as list.
This is a breaking change compared to the old Groovy implementation which relied on using a shell for calling maven.
As an example, consider this diff:
```diff
- goals: 'org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate',
- defines: "-Dexpression=$pomPathExpression -DforceStdout -q",
+ goals: ['org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate'],
+ defines: ["-Dexpression=$pomPathExpression", "-DforceStdout", "-q"],
```
Do not exit with os.Exit(1) but using log.Entry().Fatal() instead
* Golang: forward error details
* extend groovy wrapper to provide proper error message
* create closure for error handling
With the step gctsCreateRepository it is possible to create a local gCTS repository on an ABAP server
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
Also establish mavenExecute() via new JenkinsMavenExecuteRule in Groovy Unit Tests.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Extend JenkinsLoggingRule and TransportManagementServiceTest
* Print response content for status codes >= 300, also test that it's not
revealed in the case of unexpected status codes between 200 and 300
(both not included) for authentication
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
* handlePipelineStepErrors: allow disabling for stages
* update error handling in stages
* move file loading into try catch
* Cleanup DebugReport before test
Apparantly, the same DebugReport instance is loaded
during all Unit Tests. Avoid left-overs from previous
tests.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* use sonar go in groovy
* use SONAR_TOKEN
* only use owner/repo if both are set
* trim version to major version digit
* fix code climate issues
* remove Sonar defaults
* use sonar go step
* use SONAR_TOKEN
* use certs from parameters
* use docker workspace & options
* add instance parameter
* implement branchName
* implement branchName
* remove duplicate default
* update docs
* fix TODOs
* remove merge mess
* fix code climate issue
* address comments
* respect custom defaults and custom config file name
* fix typo
* remove obsolete test cases
* disable CommonStepChecks
* check step config instead of context config
* remove TODOs
* respect jenkins PR envvars
* use value from stepConfig
* Update vars/sonarExecuteScan.groovy
* rename options to config
* correct type for options
* add test cases
* log sonar.options in debug message
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Provide response from tms file upload also in case of return codes outside 2xx
* Update src/com/sap/piper/integration/TransportManagementService.groovy
Co-Authored-By: Oliver Feldmann <oliver.feldmann@sap.com>
* Update src/com/sap/piper/integration/TransportManagementService.groovy
Co-Authored-By: Oliver Feldmann <oliver.feldmann@sap.com>
* consider re-running in verbose mode only if we are not in verbose mode
* Add missing script reference when calling error
* avoid curl --fail in order to get also a response in case of 4xx 5xx
* add missing write-out
* --output instead of -o
* fix syntax errors
* fix codeclimat issue
* fix unit tests /1
* Adjust unit tests
* More unit tests
* Use other texts in verbose mode and non verbose mode in case of a failure
in order to avoid issue with hanging log messages surviving a test case (... should not be the case).
For the non verbose mode we check the http response code since there is not message we can check.
* Now with the full comment explaining the 418
* Provide different responses for verbose and non-verbose mode
in order to distinguish the cases
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Artifacts to upload are assembled for MTA projects and Maven projects with optional application sub-module. Then maven deploy:deploy-file is used as backend to upload bundles of artifacts plus sub-artifacts.
Co-authored-by: Florian Wilhelm <florian.wilhelm02@sap.com>
* provide trust store via JVM parameter
* adjust test cases
* Update vars/sonarExecuteScan.groovy
* Update vars/sonarExecuteScan.groovy
* Update sonarExecuteScan.groovy
* handle NPE
* fix Code Climate issue
* add test case
Co-Authored-By: Maximilian Lenkeit <mlenkeit@users.noreply.github.com>
* correct test case
Co-authored-by: Maximilian Lenkeit <mlenkeit@users.noreply.github.com>
* Update schema and hcp deployer version in mta.yaml file
* Add name parameter in mta yaml file
* Rename template_mta.yml to template_mta.yaml
* Fix indentation
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
* avoid sending mails to addresses which contain the string noreply
* enrich test to ensure that mails containing the string noreply never end up in recipient list
* Update MailSendNotificationTest.groovy
* inject container metadata via env vars to selenium
* set step-specific env var for uiVeri55ExecuteTests
* Revert "set step-specific env var for uiVeri55ExecuteTests"
This reverts commit 8deb39ac5b.
* remove env vars that indicate piper step
* remove env var to indicate execution through selenium step
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* remove defaulting of docker env vars
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Update vars/karmaExecuteTests.groovy
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* writeFile() cannot be passed a Map
I've changed the return type of DebugReport.generateReport() from
String to Map in order to get the generated file name as part of the
return value instead of getting it from a field of DebugReport. The
UnitTest checks whether writeFile() creates the debug_report file
successfully and whether it has the expected contents. The effect
of passing the Map instead of map.contents to writeFile() should
have been an unnecessary wrapping via Map.toString() as in the test,
but in the execution context of Jenkins, this throws an
IllegalArgumentException: Could not instantiate {... and then the
results of map.toString().
* Improve JenkinsWriteFileRule compatibility
Calling m.text.toString() is wrong, since the type stored at m.text
already needs to be a String (or GString). Expecting valid parameters
here makes sure problems are detected by tests already. (All tests
pass as before.)
* Protecode as go implementation
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
The code of extensions was not executed within the try-catch-block of
handlePipelineStepErrors. The main benefit of this change is better
logging and re-using the 'unstable' feature also for
extended/overwritten steps.
This should have been added along with DebugReport. For context:
EnvironmentUtils used to be a class alongside DebugReport and thus
there was no 'import' directive. Working with Jenkins DSL stuff
has numbed my ability to pay attention to the IDE indicating errors.
The DebugReport is a global instance where steps can store information relevant for diagnosing failed pipelines. In the SDK Pipeline, this is used to generate a debug report within the postActionArchiveDebugLog step. The reason for adding this to Piper is to feed information about extended or overwritten stages in piperStageWrapper into the DebugReport, as was done before in the SDK Pipeline's equivalent runAsStage step.
* add evaluateFromMavenPom to piper Utils
* adapt mavenExecute to accept `returnStdout` as parameter. If configured mavenExecute will return the stdout for further processing
* adapt tests of mavenExecute and mavenArtifactVersioning as well as add another exception to CommonStepsTest because mavenExecute will return a String if configured
* Fix random UUID generation in Unit Tests
* Use before and after methods
* Add import of org.junit.After
* Add go step for abapEnvironmentPullGitRepo
* Revert "Add go step for abapEnvironmentPullGitRepo"
This reverts commit 08b6b16217.
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
On groovy level we take into account:
- parameters via signature
- project config
- landscape config(s)
- for docker: nested and plain config
also more tests added ...
* Use commonPipelineEnvironment in go binary
* Update groovy part incl. tests
* Rework structure and naming
* Support influx resources in steps
* Update tests and some cleanups
* Add correct defer handling
* Address PR feedback
* Fix test
* Update resources.go
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Add new config mtarName for mtaBuild step
* Remove unnecessary whitespace changes in unit test
* Sort new config & avoid file operation when this config provided
* Modify the test to take the custom name without extension
* Update new config documentation
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* custom mta name should be given with mtar extension
* Updated the config documentation
