* changes to detectExec before master merge
* changes for detectExecuteScan
* self generated code added
* fix syntax errors and update docu
* added unit tests for fail and Group
* fix failOn bug
* add Groups as string array
* add Groups as string array
* tests and validation for groups, failOn
* Updated docs and added more tests
* documentation md files should not be changed
* Handle merge conflicts from PR 1845
* fix merge errors
* remove duplicate groups, merge error
* adding buildCode and buildTool as params
* switching build options
* building maven modules
* parameter correction
* parameter correction
* gnerate with new build parameter
* adding comments
* removing piper lib master and modifying goUtils to download 1.5.7 release
* first cleaning then installing
* multi module maven built
* multi module maven built removing unwanted code
* multi module maven built moving inside switch
* testing
* modifying the default use case to also call maven build
* modifying the default use case to also call maven build wih --
* corrected maven build command
* corrected maven build command with %v
* skipping test runs
* testing for MTA project with single pom
* adding absolute path to m2 path
* clean up
* adding switch for mta and maven and removing env from containers
* commiting changes for new detect step
* correting log message
* code clean up
* unit tests changes to detectExecute
* basic tests for new change
* restoring piperGoUtils to download correct piper binary
* code clean up
* code clean up
* revert to clean version
* add unmap parameter to detect
* Added Additional parameters for BlackDuck scan
* Added detect tools paramater
* fix detect.yaml to accept correct data type
* fix codeclimate issue in detect.yaml
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* [refactoring] Make room for other upload action: move package
In the near future we will have more upload actions, like
SOLMAN, RFC. Here we prepare the package structure for that.
* don't use aliasing
* rename entities (no leading CTS)
From the current location inside "cmd" the npmExecutorMock cannot
be used from any coding inside "pkg".
When we would like to re-use the npm functionality we have also to
provide tests and this requires having a mock.
In order to be able to use the mock from pkg we move the mock from
"cmd" to "pkg" into package "npm".
With the shift from package "cmd" to "npm" a lot of fields in the mock
has been made public.
* Upload in Chunks
* fix unit tests for file upload in chunks
* upload in chunks review round 1
* Upload in Chunks - review round 2 - comments
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* initial commit of yaml file
* initial commit for HaDoLint in GO
* add helper function to load file from url
* load config file
* write report information to disk
* comment the code
* refactor groovy code
* remove download function from FileUtils
* use http.Downloader
* rename step files
* update generated files
* update generated files
* remove duplicate commands
* add credentials for config url
* add generated test file
* reuse piperExecuteBin functions
* correct step name
* update go step
* deactivate test
* fix import
* use differing go step name
* rename step
* correct result publishing
* correct command name
* expose tls insecure flag
* hand through error
* disable tls verification
* fix tls disabling
* use credentials
* mow
* reformat
* add qgate only if set
* correct report name
* remove old defaults
* add qgate to defaults
* handle report name
* restore default
* remove unused step config
* use piperExecuteBin
* remove obsolete type
* add test cases
* remove groovy tests
* move client parameter handling to run function
* use custom interfaces and mockery
* remove commented code
* correct struct names
* rename parameter dockerfile
* add further asserts
* cleanup
* change file permission to read/write
* remove tokenize
* add further comments
* init http client only if necessary
* add todo
* Revert "rename parameter dockerfile"
This reverts commit 2a570685b8.
* add alias for dockerfile parameter
* correct test case
* Apply suggestions from code review
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add comment about mock assertions
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* expose GetExitCode
* set error category for user issues
* fix typo
* Update command.go
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* fix checkmarx docs
* go generate
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Is there any benefit from having
```
assert.Error(./.)
assert.EqualError(./.)
```
?
assert.Error ensures that we have an error.
assert.EqualError ensures that we have an error and
moreover it checks for a specific error. Hence
assert.EqualError does all and more what assert.Error
does.
In case there is a benefit from that pattern this PR should not be merged.
In case there is not benefit from that pattern we should abandong that pattern.
* new 'keepFailedDeployments' parameter which removes the --atomic flag
* Update resources/metadata/kubernetesdeploy.yaml
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* go generate
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Make sure the UA scan is known to the scan object. Fixes downloading reports later on.
* Move polling into pkg/whitesource, add test for e2e scan
* Remove conditions from stash config resource
* Don't use version stored in CPE. This will prevent the versioningModel from being applied.
This change extends the npmExecuteScripts step to support execution of
npm scripts for specific modules. Previously, it was not possible to
execute npm scripts only for specific modules. Now, if the parameter
buildDesriptorList is set the scripts defined by the runScripts
parameter will be executed for the modules defined by
buildDescriptorList. Note, in this case the buildDescriptorExcludeList
will be ignored.
* makes containerImage not mandatory
* Adds kubectl container
* Adds log statement to debug
* adds general to container image
* removes GENERAL again
Removes condition from Kubectl container
* removes workDir
* marks logs as debug
* adds workingdir again
* Adds author to commits
* Adds commit time now
* remove deprecated and reorder
* adds deprecated again to containerRegistryUrl
Adds GENERAL scope to containerImage
* updates generated file
* Renames containerImageNameTag
* adds else case
* adds debug log
* code cleanup
* adds debug log
* revert
* adds debug logs
* revert
* makes root path not hidden
* revert
* Read container properties
* Removes debug message
* Removes debug message
* Removes general scope again
* Fixes unit test
* Adds helm capabilities to the gitopsUpdateDeployment step
* Adds helm capabilities to gitopsUpdateDeployment step
* Removes condition from input field
* Adds test for invalid deploy tool
* Fixes typo
* Adds tests for git errors and file errors
Simplifies test setup
* Adds test for error on image name extraction
* fixes URL variable name
* adds workind directory to paths
* Refactors too long method
* Reverts refactoring method
* Adds repository name as parameter
* Adds glob method
* Test glob method
* Revert "Test glob method"
This reverts commit ac11b54c14.
* Revert "Adds glob method"
This reverts commit ddf47ddebe.
* Revert "Adds repository name as parameter"
This reverts commit 8fc471c909.
* Removes getWd
* Adds stash deployDescriptor
* removes = from paramters
* Revert "removes = from paramters"
This reverts commit 3ecb3665e2.
* Adds " around parameters
* adds logging of all files
* Updates helm to version 3.3.4
* Clean up debug logs
* Raise error if no branch name provided.
Defaulting should be handled by step configuration.
* clean code
* Updates fields and adds checks for required field for certain deploy tools
* Fixes default commit message
* Update long description
* Removes default parameter
* Update resources/metadata/gitopsUpdateDeployment.yaml
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Updates yaml file
* Add error category and removes too much wrapping
* Update generated file
* Checks all parameters before returning the error
* Introduces constant
* Renames constant
* Fixes unit tests
* unexpose constants
* Makes tests thread safe and resilient to failed deletion
* Remove methods that did not work properly with hash containers rather than tags.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Should avoid issues with this file being owned by root (perhaps via running in docker container), preventing the workspace from being cleaned properly.
* makes containerImage not mandatory
* Adds kubectl container
* Adds log statement to debug
* adds general to container image
* removes GENERAL again
Removes condition from Kubectl container
* removes workDir
* marks logs as debug
* adds workingdir again
* Adds author to commits
* Adds commit time now
* remove deprecated and reorder
* adds deprecated again to containerRegistryUrl
Adds GENERAL scope to containerImage
* updates generated file
* Renames containerImageNameTag
* adds else case
* adds debug log
* code cleanup
* adds debug log
* revert
* adds debug logs
* revert
* makes root path not hidden
* revert
* Read container properties
* Removes debug message
* Removes debug message
* Removes general scope again
* Fixes unit test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* add vaultSecretFileReferences
* fix test
* fix test
* go generate
* remove code duplication
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* Adds piper step to update deployment configuration in external git repository.
https://github.wdf.sap.corp/ContinuousDelivery/piper-ita/issues/21
* Adds handling of branchName as an optional parameter
* Update resources/metadata/gitopsUpdateDeployment.yaml
Feedback about description
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Adapt to interface guide
* Refactors to GitopsExecRunner
* Refactors to GitopsExecRunner in test
* Removes unnecessary mocked methods
* Adds tests for git utils
* Adds new step to CommonStepsTest.groovy
* Updates description from yaml
* Restricts visibility of methods and interfaces
Adds comments where necessary
* Updates comments
* Fixes URL name
* updates description
* updates generated file
* Fixes compile issue in CommonStepsTest.groovy
* Updates long description
* Updates test to run green on all kind of OS
* Removes global variables from tests
* Default branch: master
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Typo in Hierarchy
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Refactors test to allow parallel execution
* Renames utility variable in gitopsUpdateDeployment.go
* Renames error variables in gitopsUpdateDeployment.go
* simplified parameters for kubectl
* Refactors util classes to use parameters rather than global variables
* makes username and password mandatory
* remove unnecessary mandatory flag
* remove new methods from mock that are not necessary
* replaces with EqualError
* replaces with NoError
* update generated file
* refactor tests
* refactor tests
* make tests parallel executable
* parallel execution of tests
* Refactors interfaces to stop exposing interfaces
* Feedback from PR
* Simplifies failing mocks
* Renames variables and interfaces
* Fixes error messages
* shorten variable names
* Renames unused parameters in tests
* Cleanup nil parameters
* Typo
* Wrap errors and remove unnecessary logs
* Remove containername and filePath from GENERAL scope
* correct generated file
* corrects expected error messages
Co-authored-by: OliverNocon <oliver.nocon@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
This change fixes an issue with the default maven excludes where the step would fail when performance tests are present in the project, but are not a module in the root pom. In addition, it provides a small fix of a method where a wrong path was passed to an error.
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* docu update for issue 2157
docu update for issue 2157
* Docu updates for issue #2157 take 2
* Docu updates for issue #2157 take 3
* Docu updates for issue #2157 take 4
* add type to sonar field
* respect type of influx fields
* update generated code
* switch type
* copy changes from #1885
* log JSON data
* read simple values from json
* Update InfluxData.groovy
* Revert "Update InfluxData.groovy"
This reverts commit c8cfdf381f.
* Revert "read simple values from json"
This reverts commit 94b69866d2.
* Revert "copy changes from #1885"
This reverts commit 2471b4475e.
* update TODO
* make dockerUsername secret
* regenerate sources
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* handle non-string values as JSON
* change value type to interface in resources
* regenerate code
* add test cases
* handle reading of json files
* write json data to json files
* fix assignment
* use GetResourceParameter
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* align parameters for Detect
* include feedback
* update generation and formatting
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Update fortifyExecuteScan.go
* Update fortifyExecuteScan.go
* Docs are lying
Checked the API which returns a status similar to that of artifact
* Update fortifyExecuteScan_test.go
Vars file handling centralized
We have the same coding for handling varsf-files and vars. With that change we shift to having one common coding for that
* adding defult parameter to exclude test to detectExecute yml
* generating the modified cmd/detectExecuteScan_generated.go
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
This change fixes the step mavenExecuteStaticCodeChecks to skip execution
if no tool is configured. Previously, the step was executed even when both
supported tools, i.e., PMD and SpotBugs were disabled.
* Add projectKey and coverageExclusions params
* Also add binary, coverage exclusions and jacoco related options to sonar execution.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
Co-authored-by: Kevin Hudemann <kevin.hudemann@sap.com>
* fix redundant type issues
* cleanup
* extract report function for protecode package
* use speaking status constant for API results
* remove unconsidered return value
* correct switch statement
* handle severe vulnerabilities
* Apply suggestions from code review
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* correct test name
* return errors from WriteReport function
* expose ReportData struct
* set Error Category
* refactor constant visibility
* change type name
* describe type
* change type name
* fail after report generation
* do not fail on report write errors
* add error as return value
* fix typo
* use require statements
* assert major vulnerabilities
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add trustStorePassword to SONAR_SCANNER_OPTS
-Djavax.net.ssl.trustStorePassword=changeit has to be provided to the SONAR_SCANNER_OPTS otherwise we get a
ERROR: SonarQube server [https://sonar.wdf.sap.corp] can not be reached on our new jenkins server
Caused by: javax.net.ssl.SSLException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
we tried to inject via "options" variable of the sonarExecuteScan task, but seems it is not recognized as it is overwritten by the SONAR_SCANNER_OPTS defined in this file.
* adapt tests
* Update sonarExecuteScan.go
* Update sonarExecuteScan_test.go
* Update sonarExecuteScan.go
* Update sonarExecuteScan_test.go
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
stageName from parametersJSON was ignored up until now and there was a TODO in the code. The precedence from the TODO is not clear to me. It said "first env.STAGE_NAME, second: parametersJSON, third: flag". So maybe I implemented it wrongly, but it would be easy to change. Right now, STAGE_NAME is always in the ENV when piper is called from Jenkins, but to mimic the behavior of being able to pass `stageName` in the parameters and override the ENV value, this is what happens in go as well now.
The reason why `stageName` needs to be evaluated before other step parameters is that it affects config evaluation. So I've added a function which is called first in `PrepareConfig()`.
The SAP NPM registry has been migrated to the default public registry,
thus the separate configuration with the sapNpmRegistry is not required
anymore.
All packages from npm.sap.com have been migrated to npmjs.org
and in the future SAP packages will only be available from the default
public registry.
* log more information about docker credentials
* remove println
* fix format issues
* Update protecodeExecuteScan.go
* correct weird space characters
* add test case for correctDockerConfigEnvVar
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Adapt changes of abautils pkg
* Add test for polling
* Minor fix
* Fix yaml spacing
* Add longdescription to yaml
* Refactor abaputil methods
* Refactoring
* Refactoring
* Minor fix
* Minor fixeds
* Adapt to new abaputils.AUtilsMock
* Delete obsolete initial checks for params
* Fix manageGitRepoUtils_test.go
* Adjust pollEntity tests
* Add abapEnvironmentCheckoutBranch.groovy file
* Add documentation for abapEnvCheckoutBranch step
* Add step to whitelist in commonStepsTest.groovy
* Add GENERAL scope to host parameter
* Fix generated.go file
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Adapt changes of abautils pkg
* Add test for polling
* Minor fix
* Fix yaml spacing
* Add longdescription to yaml
* Refactor abaputil methods
* Refactoring
* Refactoring
* Minor fix
* Minor fixeds
* Adapt to new abaputils.AUtilsMock
* Delete obsolete initial checks for params
* Fix manageGitRepoUtils_test.go
* Adjust pollEntity tests
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
Provide cloudFoundryDeploy step in GO layer.
Groovy part untouched. Groovy-Stub needs to be provided later (with a feature toggle in order to be able to switch back)
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* kubernetesDeploy: add step configuration alias
This is used for backward compatibility
* fix commonPipelineEnvironment references
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* If a step declares a parameter of type string, depending on how the config is written, it is no longer ignored, if it is interpreted by the yaml parser as integer or float value.
* If an expected parameter is present in the configuration, step execution will consistently fail if the parameter has the wrong type, no sensible conversion can take place, and it is known that the parameter will be ignored.
* For all type-mismatches that have no implemented conversion, a warning is logged. (It isn't known whether the conversion actually works, since it depends on both the yaml and json packages and future changes there.)
* Entries in the evaluated config with a value of nil are ignored.
* changes to detectExec before master merge
* changes for detectExecuteScan
* self generated code added
* fix syntax errors and update docu
* added unit tests for fail and Group
* fix failOn bug
* add Groups as string array
* add Groups as string array
* tests and validation for groups, failOn
* Updated docs and added more tests
* documentation md files should not be changed
* Handle merge conflicts from PR 1845
* fix merge errors
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* removed whitespaces in filePatterns and add zip file count log
* safer string-replace for whitespaces
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
This change adds a buildDescriptorExcludeList parameter to
npmExecuteScripts, to enable the exclusion of certain directories when
executing npm scripts. Previously, npmExecuteScripts could only execute
scripts in all packages.
Now it is possible to provide paths or patterns as elements of the
buildDescriptorExcludeList to exclude packages when executing npm scripts.
o Up to now we work on a private and shared instance of command.Command inside
the cloudfounrdy package. We need to be able either configure this instance
(environment variables) according to the use case. One option is to hand over
an already configured instance which is used elsewhere. This is what we do with
this commit.
o With this commit we remove the instance which is shared within the cloudfounrdy
package and to provide an instance with a receiver which gets handed over to the
functions. Hence we are thread save: parallel invoctation of e.g. Login will not
affect each other.
* [refactoring] move the shell/command related interfaces into pkg/command
otherwise we are not able to use the corresponding mocks for the items contained in pkg since
these interfaces are not visible from the pkg folder
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Whitesource MVP for Gradle, Golang, and NPM/Yarn
* Refactoring
* Refactor and cleanup, better error checking
* publish stepResults, use pkg/versioning, bubble up errors, add gomod versioning support
* Run gofmt and cleanup comments
* Resolve PR comments
* Update resources/metadata/whitesource.yaml
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Only determine project coordinates if they are missing
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Gradle versioning artifact
* fix gradle artifact version regexp and refactor
* Fix token extraction from output buffer
* Fix some issues with pip and jsonfile versioning logic
* Remove useless spacing
* Remove unnecessary test file and fix naming style for JSONDescriptor
* Automatically download wss-unified-agent if file does not exist
* adds downloadVulnerabilityReport, checkSecurityViolations, minor refactoring
* adds config.ReportDirectoryName, improves readability
* Version-wide reporting for vulnerabilities and list of libraries.
* Refactor and improve build accuracy
* fix sed command
* Add includes file pattern config option
* Adds --exclude command line flag
* run go mod tidy and regenerate step framework
* Fix unit tests
* revert changes
* poll project status before downloading reports
* merge with master
* go mod tidy, go fmt, and fix whitesource unit test
* sync go.mod
* sync go.mod again
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Fix alias cfServiceKeyName
* Adapt documentation
* Fix alias use in go files
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* Add error category parsing to cmd execution
It is now possible to define `ErrorCategoryMapping` as a `map[string][]string` on a `Command`.
The format contains the category as key which has a list of error patterns assigned.
Example:
```
cmd := Command{
ErrorCategoryMapping: map[string][]string
"build": {"build failed"},
"compliance": {"vulnerabilities found", "outdated components found"},
"test": {"some tests failed"},
},
}
```
Setting this map triggers console log parsing when executing a command.
If a match is found the error category is stored and
it will automatically be added to the `errorDetails.json`.
* clean up go.mod
* fix test
* fix test
* Update DEVELOPMENT.md
* fix tests
* address long console content without line breaks
* scan condition update
* fix test
* add missing comment for exported function
* Update pkg/command/command.go
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Ensure npm dev dependencies are available after mtaBuild as they are required by certain tests.
Co-authored-by: Kevin Hudemann <kevin.hudemann@sap.com>
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
This change adds linting support for Java-/TypeScript projects based on the checkDefaultLint and checkUserLint steps from Cloud SDK Pipeline.
Following options are supported:
1. Define a linting script named ci-lint in the package.json file(s) of the project
2. Provide a configuration for ESLint as part of the project
3. If none of the above: ESLint is executed with a general purpose configuration.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Add possibility to add category to failures
It is now possible to set the error category within the flow.
When exiting the program the error category can be used.
There is a convenience function available for exiting with a previously set category,
for example
```
log.SetErrorCategory(log.ErrorCompliance)
...
log.FatalError(err, "configuration error")
```
* extend test
* go mod tidy
* add missing comment
* update information about error categories
* Update DEVELOPMENT.md
This change refactors the npm pkg and npmExecuteScripts implementations
to be reusable for future steps, e.g., npmExecuteLint.
In addition, it fixes few small bugs related to unit test execution on
Windows and the fileUtils mocking implementation.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Extend mta build step to install maven artefacts after build to allow re-using them in later stages (additional unit or integration tests which might not be running as part of the "build" life-cycle).
* The metadata for artifactPrepareVersion-go specifies a container for when the buildTool is maven.
* The alias to 'mavenExecute' was removed. The problem with this is that when a section containers is included in the metadata, dockerImage will always be picked up from mavenExecute, the conditional dependency on buildTool will not even be considered. Parameters such as m2Path, projectSettingsFile and globalSettingsFile should be configured in general/maven if necessary.
* When the step ends up being executed within dockerExecuteOnKubernetes, we need to preserve the .git folder. This folder would normally be excluded by the default excludes of the stash step. There was already a comment that suppressing this behavior by passing useDefaultExcludes: false was problematic (unfortunately without going into details), so I've added a new parameter to dockerExecute and dockerExecuteOnKubernetes named stashNoDefaultExcludes (note the reverted meaning to ease preserving the default behavior when this parameter is not provided). This parameter is passed to piperExecuteBin from the artifactPreferVersion groovy wrapper.
Provide a timeout parameters to malwarescan step. This is forwarded to the piper http layer. The default used there is 10 seconds with is not useable for that use case for larger files.
* Update fortify.yaml
* src, exclude and pythonAdditionalPaths are now lists of strings
* Re-implement pythonIncludes and pythonExcludes as aliases of src and exclude
* Fix using the correct separator (; on windows, : on everything else)
* Tokenize also python "includes"
* mvnCustomArgs was removed
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* also reduce code duplication in token fetching
* concatenate classpaths from multi-maven projects
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Kevin Hudemann <kevin.hudemann@sap.com>
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* piperExecuteBin - prevent non-speaking exit code 1 error
Make sure that error during context resolution gets more context.
* properly provide golang errors
* improve error message, when no details are available
* Add error category
* Update getConfig.go
* streamlined error category with standard term
* remove unused import from test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* added new step gctsDeployCommit
* suggested PR fixes applied
* fixed test
* Remove unused imports
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* added URL encoding for 'request' parameter
* regenerate after change
* minor changes
* renamed step to 'gctsDeploy'
* changed name
* remove space
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Create step to save container image
* add error handling for directory creation
* Update permissions
* remove indirection and add tests
* finalize tests
* Fix test to also run on linux environments
* Update cmd/containerSaveImage.go
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Update resources/metadata/savecontainer.yaml
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Update cmd/containerSaveImage.go
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Update cmd/containerSaveImage_test.go
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Update generated file
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Enable multiple repositories
* Improve readability
* Adapt documentation
* rerun generator
* Add initial info line
* Improve logging
* Enable configuration in GENERAL section
* Adapt yml abapEnvironmentPullGitRepo
* Add aliases for crendetialsId
* Change order of credentialsId aliases
* Change prio of different names of credentialsId
* Add docu to sidebar
* write flag for sonar execution to influx
* add missing step metadata file
* first attempt to read influx from disk
* add missing import
* correct glob pattern
* use file path
* correct type names
* cleanup
* fix code climate issue
* fix typo
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Add parameter "--ignoreCustomDefaults"
* Pass to piper customDefaults from config also via --defaultConfig
... and add "--ignoreCustomDefaults".
* Log output when ignoring customDefaults
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* artifactPrepareVersion: additional options
it is now possible to use
* a Unix timestamp number
* a short commit hash
* Add dedicated Docker versioning scheme
Avoid maven error `Unknown lifecycle phase \"-\"` when the value of a define contains `-`.
Don't split and trim maven arguments. Expect they come in as a list, keep them as list.
This is a breaking change compared to the old Groovy implementation which relied on using a shell for calling maven.
As an example, consider this diff:
```diff
- goals: 'org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate',
- defines: "-Dexpression=$pomPathExpression -DforceStdout -q",
+ goals: ['org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate'],
+ defines: ["-Dexpression=$pomPathExpression", "-DforceStdout", "-q"],
```
* fix docu
* enhanced step docu and metadata
* regenerate after changing metadata
* small wording change
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
Do not exit with os.Exit(1) but using log.Entry().Fatal() instead
* Golang: forward error details
* extend groovy wrapper to provide proper error message
* create closure for error handling
This change enables the use of the artifactPrepareVersion go step instead
of the deprecated artifactSetVersion groovy implementation for the Cloud
SDK Pipeline. The groovy wrapper for the artifactPrepareVersion step now
enables the use of the download cache and minor issues are fixed.
With the step gctsCreateRepository it is possible to create a local gCTS repository on an ABAP server
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
Also establish mavenExecute() via new JenkinsMavenExecuteRule in Groovy Unit Tests.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Extends kubernetesDeploy step to support Helm 3
Currently, the kubernetesDeploy step has no support to Helm 3 due to the fact that:
- the initialization command used works only for Helm 2
- the image used when running the helm CLI is based on Helm 2
The need for Helm 3 support comes from the fact that Helm 3 introduces major architectural changes,
more specifically, the removal of its server-side agent called Tiller - thus, being incompatible with
one another.
This commit adds this support by introducing a new configuration field (helmVersion).
By default, its values is set to 2 (Helm 2) to avoid breaking any existing functionalities.
* Use deployTool field to decide between Helm 2 or 3
* Remove helm init and replace wait for atomic in v3
* Update cmd/kubernetesDeploy.go
Nice catch!
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Add documentation for kubernetesDeploy step
* Add helm3 example for kubernetesDeploy step using mandatory fields
* Add new line at the end of kubernetesDeploy documentation
* Link kubernetesDeploy step with docs generator
* Add possible values for deployTool in kubernetesDeploy
* dummy change
* Revert "dummy change"
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* sonar: fix test cases
some unit tests uses path.Joint() instead of filepath.Join()
as the code does. on window these unit tests fail.
* missing newline
* Update cmd/sonarExecuteScan_test.go
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Changed from passing config as an array that was implemented badly to correct implementation that JSON config gets passed with ' '
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* use sonar go in groovy
* use SONAR_TOKEN
* only use owner/repo if both are set
* trim version to major version digit
* fix code climate issues
* remove Sonar defaults
* use sonar go step
* use SONAR_TOKEN
* use certs from parameters
* use docker workspace & options
* add instance parameter
* implement branchName
* implement branchName
* remove duplicate default
* update docs
* fix TODOs
* remove merge mess
* fix code climate issue
* address comments
* respect custom defaults and custom config file name
* fix typo
* remove obsolete test cases
* disable CommonStepChecks
* check step config instead of context config
* remove TODOs
* respect jenkins PR envvars
* use value from stepConfig
* Update vars/sonarExecuteScan.groovy
* rename options to config
* correct type for options
* add test cases
* log sonar.options in debug message
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* iterate over found pom.xml files for maven projects, deploy found artifacts
* classifiers are found by testing prefix/suffix of artifact
* Remove additionalClassifiers option
* Improve documentation
* Fix fallback for finalBuildName
* Ignore mvn modules with packaging != pom but no target dir
* Fail when main artifact is missing, add more tests
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Fix typos
* Support aliases also for secrets
* Adapt & extend Unit Tests
* Output deprecation warning for param/secret aliases
... if the alias is marked as 'deprecated'.
* Config and Defaults via http(s)
Add the possibility to use http(s) urls as path to
* config.yml
* defaults
* Remove comment
* Addressing PR feedback
* Fix error handling
* Replace the default maximum request deadline with a default timeout on the transport level.
* Keep the possibility to set a maximum request deadline.
Artifacts to upload are assembled for MTA projects and Maven projects with optional application sub-module. Then maven deploy:deploy-file is used as backend to upload bundles of artifacts plus sub-artifacts.
Co-authored-by: Florian Wilhelm <florian.wilhelm02@sap.com>
* Golang step metadata: Config aliases for steps
This will ease following scenarios:
* config migration due to step name changes
* re-use of more general config, e.g. `mavenExecute` in `mavenBuild`
* fix CodeClimate finding
* Fix panic if original stage config does not exist yet
* add static code checks for maven based projects as:
* pmd plugin
* spotBugs plugin
* test modules as unit-tests and integration-tests will be ignored by default. Additional modules to ignore are configurable
* for pmd: rulesets and excludes are configurable
* for spotBugs: includeFilter and excludeFilter are configurable
ExecMockRunner and ShellMockRunner both needs an environment. "Extending"
here leads to "subclasses" for both cases. That is more long-winded since
it could be.
I don't understand why there should be a two dimensional array.
When dealing with envs we have normally a list containing entries like
[]string{"DEBUG=true", "HOME=/home/me"}
Having two dimensional env arrays would mean to have several alternate
environment in the tests at the same time. Don't think there is a need
for that.
* same behaviour for shellRunner and execRunner wrt errors and stdout
* replace shouldFail with shouldFailOnCommand
* [formatting only] format struct
* Move to regex for execptions and stdout
* shrink code
* Protecode as go implementation
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Support yml and yaml extension of config files
* Update cmd/piper.go
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Enhance telemetry reporting
* Use central telemetry data object
* Add duration
Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Use commonPipelineEnvironment in go binary
* Update groovy part incl. tests
* Rework structure and naming
* Support influx resources in steps
* Update tests and some cleanups
* Add correct defer handling
* Address PR feedback
* Fix test
* Update resources.go
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Remove redundancy: no duplicate stepName
Up to now: provided explicity via command line argument and also contained in the
step metadata (which are as a file also provided as command line argument).
Now it is retrieved from the metadata file.
* Step name is not a required parameter anymore
Since the step name is derived from the config file
* Adjust method signature in test
* Read the project config only if it exists
This avoid trying reading the file and have the control flow based
on errors. Beside that it helps troubleshooting when we have some
logging (debug level only).
* formatting only
* Adjust log level
There was some command parsing with failure in case it started with fail. That is
IMO less transparent. Now we prepare more explicit with a failure from outside. This
enables us to prepare an error like we expect it in the free wild.
* Export general configuration - part 2
First part in #956 missed to export the elements of the struct ...
* Add comment for exported struct
* Add function for opening config files
