* fixed generated output parameters for influx
* change name to lower case
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Add Changes for value of docker image
* Get docker image value
* Fix
* Fix unit
* Add chnages for kaniko and mta builds
* Fix
* Test changes
* Test
* Move func ResolveMetadata to stepmeta.go
* Fix
* Change getConfig.go
* Fix getting docker value for mta, npm and kaniko
* Fix according to suggestions
* Add func to get only value of docker image
* Test empty value of docker image
* Fix for getDockerImageValue
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Removes unecessary fields from telemetry, restructuring splunk pkg
* Removes t.skip() and reactivates integration test
* Adjusts tests for fatalHook and helper functions, including log test
* Moves pipelinetelemetry to inner source, removes pipelineTelemetry from telemetry pkg, using generic map[string]interface for splunk
* Removes Read JSON from fatalHook -> moves to inner source
* Removed log output test
* go fmt
* log step telemetry data send to splunk
* Adjusts error logging
* Adds log information in case api information could not be retrieved
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetRequest function which holds the response in memory (not saved to disk)
* Implements GetLog() function for ADO, adds function to read PipelineRuntime
* PAT has been revoked
* Changes http package, s.t. if password only is required basic auth works too
* Adds env variable for azure token, error handling in case of unauthenticated/nil response
* Adds logging output in case env variable can not be read and fallback variable needs to be used
* Adds usage of environment variables for auth, uses jenkins api
* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function
* Adds initaliziation function for orchestrator authetnication
* Adds settings struct for orchestrator authentication
* Adds function to whole logfile to Splunk
* Struct for pipeline related telemetry information
* Increase messagebatch size to 10k
* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk
* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation
* Changes content type to txt
* Send telemetry independent of logfiles, increases amount of messages per file
* Adds JobURL for orchestrators and UnknownOrchestrator as fallback
* telemetry makes use of orchestrator specific information
* Adds orchestrator independent correlationID
* Adds custom fields for pipeline status
* go fmt
* Removes env var test - no env variables are read anymore
* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized
* Removes Custom fields from telemetry as these can not be reflected in SWA
* Adds custom telemetry information (piperHash,..) to each step telemetry information
* Removes falltrough in case no orchestrator has been found
* Updates tests for orchestrator package
* Adds orchestrator import in generated files
* Updates generator files for internal library
* Adds orchestrator telemetry information to steps
* Updates generated files, fatalHook writes to cpe
* Go generate from master, go fmt
* Adds Custom Data field LastErrorCode
* Removes GetLog() test
* Update init_unix.go
* Update docker_integration_test_executor.go
* Update integration_api_cli_test.go
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Renames customTelemetryData to stepTelemetryData
* Adjustments to orchestrator-package, cleanup, adds JobName
* Adjusts commonPipelineEnvironment path
* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package
* Go fmt
* Changes path for errorDetails, adds debug information
* Removes custom fields from step, adds orchestrator, commithash to baseMetadata
* Adjusts tests for telemetry package
* Adds tests for orchestrator
* Updates generated files, initalization of splunk client only if its available in the config
* Fixes typo in helper go
* Update pkg/http/downloader.go
* Update pkg/http/downloader.go
* Update pkg/log/fatalHook.go
* Update fatalHook.go
* Update pkg/splunk/splunk.go
* Update pkg/telemetry/data.go
* Adds GetBuildStatus() and GetAPIInformation() to orchestrators
* error formatting
* Bugfix: dont send telemetry data if disabled, adjusts test
* go fmt
* Use correct error handling
* Update pkg/telemetry/telemetry.go
* Fixes telemetry disabled in the tests
* Fixes http tests
* Log fatal errors to logFile
* Adds CustomReportingConfig to hooks
* Cleanup comments in splunk package
* Adds possibility to send telemetry to custom endpoint
* Adds debug output for the payload
* Debug output for the payload as a string
* Adds test cases for changes in telemetry package
* go fmt
* Adds generated files for new step
* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17
* Adjusts test for sonarExecuteScan
* Adjusts test for sonarExecuteScan
* Adds explanation for customreportingConfig
* Makes disableing of customSend more obvious
* Adds custom step reporting to each step, updates generated files, adjusts helper testdata
* fixes unit test wrong usage of logging
* Send pipeline data altough there has been no error, adjust test cases
* Reverts changes for customReporting
* Updates generated files, removes customReporting
* Removes writing errorDetails to CPE
* Reverts usage of customreporting
* go fmt
* reverts changes in http_test
* reverts changes in http_test
* Skips integration cnb test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* feat: first parts of new run struct
* add parts for new stage condition handling
* update conditions
* feat: finalize conditions and tests
* feat(checkIfStepActive): support new CRD style conditions
* feat(docs): allow generating stage docs
* chore(docs): make step directory configurable
* fix: tests
* add option to output file
* Update checkIfStepActive_test.go
* create build settings for maven
* cases for when mavenBuild may be present
* fixing unit test for mavenBuild to include cpe
* changing position of buildSettngsJson to be called atfter build runs
* package
* extending the struct for other build types
* adding values for mta build settings
* changing config data type
* adding npm build settings
* unit tests
* fix trailing space
* typo correction in yaml
* Vitalii/build settings info (#3277)
* Add buildsettings package
* Improve buildSetting package for mta, npm
* Add unit-test
* Fix
* Fix
Co-authored-by: Vitalii Sidorov <vitalii.sidorov@sap.com>
* review changes
* removing buildTool param
* changing npm script name
* fix npmExecute tests
* including build settings info in npm struct
Co-authored-by: Your Name <you@example.com>
Co-authored-by: kingvvgo <56587879+kingvvgo@users.noreply.github.com>
Co-authored-by: Vitalii Sidorov <vitalii.sidorov@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Fixed validation for possibleValues option
* Change oneof-custom to possible-values
* go generate
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* new checks for commitIDs
* new checks for commitIDs
* relocate step from build stage to initial checks + refac
* log list
* fix log + check
* log format
* fix unit tests
Co-authored-by: Christian Luttenberger <42861202+bluesbrother84@users.noreply.github.com>
* empty cpe values from disk and cpe map on condition
* changing log level
* changing log level from info to edbug
* changing empty logic for empty string
* adding toBeEmptied condition
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* [WIP] bindings support for cnbBuild step
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
* add unit tests
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
* switch to mapstruct for more meaningful errors
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
* add integration test for bindings
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
* Add documentation for cnbBuild.bindings
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
* fixed unit tests
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
* apply codeclimate suggestions
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
* renamed field "secret" to "key"
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
* Adds retry for whitesource download in case of 404 not found
* Adds retry mechanism for 'forbidden errors' which occasional happen when downloading the unified agent
Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>
confirm no longer done based on package status but now based on boolean indicator which is set during assembly step.
Thus confirm can now be placed after release packages.
* changes to detectExec before master merge
* changes for detectExecuteScan
* self generated code added
* fix syntax errors and update docu
* added unit tests for fail and Group
* fix failOn bug
* add Groups as string array
* add Groups as string array
* tests and validation for groups, failOn
* Updated docs and added more tests
* documentation md files should not be changed
* Handle merge conflicts from PR 1845
* fix merge errors
* remove duplicate groups, merge error
* adding buildCode and buildTool as params
* switching build options
* building maven modules
* parameter correction
* parameter correction
* gnerate with new build parameter
* adding comments
* removing piper lib master and modifying goUtils to download 1.5.7 release
* first cleaning then installing
* multi module maven built
* multi module maven built removing unwanted code
* multi module maven built moving inside switch
* testing
* modifying the default use case to also call maven build
* modifying the default use case to also call maven build wih --
* corrected maven build command
* corrected maven build command with %v
* skipping test runs
* testing for MTA project with single pom
* adding absolute path to m2 path
* clean up
* adding switch for mta and maven and removing env from containers
* commiting changes for new detect step
* correting log message
* code clean up
* unit tests changes to detectExecute
* basic tests for new change
* restoring piperGoUtils to download correct piper binary
* code clean up
* code clean up
* protecodeExecuteScan :: fixing file upload for binaries
* protecodeExecuteScan :: fixing protecode generate file
* Fix upload test
* protecodeExecuteScan -> fixing tests
Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* support for untar in fileutils
* handeling strip component level like tar cli
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
* Implemented validation for the option possibleValues
* Has been added the option mandatoryIf to config with validation
* Fixed issues found during code review
* improved golang template
* Fixed tests. Added validation for mandatoryIf option
* Fix typo
* Fixed tests
* Validation was refactored. Added options
* Added default value for parameters with possibleValues option
* Validation was moved after the configuration resolution
* Canceled some default values
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* extend orchestator to provide stage name
* use orchestrator specific stage name
* fix test case
* remove comment
* fix test case
* prettify
* change something..
* do not exit
* Update pkg/orchestrator/azureDevOps.go
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* mta build config
* http request to upload mtar
* adding basic auth
* using put
* test file name
* hard coding the mta org and artifact is
* new version
* new version
* mtar group
* errors
* better error message
* log info
* log info
* correct mtar artifact name
* adding teh correct name
* test
* name changes
* clean up
* changing mtarVersion to version
* changing artifact name
* forcing release build
* forcing release build
* force profile
* force profile
* force profile
* force profile
* force profile
* force profile
* force profile
* force profile update
* force profile update
* profile update
* debug
* debug
* debug
* debug
* rewrite xml update
* rewrite xml update
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* unmarshal solution
* outputin publish repo url
* removing fetch coordinates condition
* checking settings xml
* fixing artifact id cpe
* release artifact, package and group to cpe
* including versioning type as a cpe
* creating new settings xml file
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* creating parent folder
* changing to project settings
* function name change
* using glbl settings xml
* modiying the npm settings
* modiying the npm settings
* modiying the npm settings
* modiying the npm settings
* using file path join for m2 settings file
* generator
* unit tests
* hardening error message
* removing versioningType
* removing versioningType
* new vault profile paths
* error message improvement
* unit test fixes
Co-authored-by: Your Name <you@example.com>
* add policy status reports
* add policy status and cumulus json
* update projectver link + test
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* modify wrapper http to hold ca rot
* modifying the http client for maven build
* adding checks
* eliminating seperate jks
* test tls true
* insecure flag remove
* error debug
* storing cert
* sap root cert
* error
* only child cert
* test
* maven test
* moving outside loop
* changing pointer
* dont download existing certs
* typo fix
* removing mavenBuild test
* code clean up
* making hadolint using always cert true
* custom tls link for hadolint trust
* error handel
* extended condition for modifying custom tls
* unit test case
* checing when to add the customLinks
* not breaking existing hadolint client config
Co-authored-by: Your Name <you@example.com>
This follows up on #3024
Setting emptyValue to s.th. like `--entrypoint=''` will break in case the argument is properly escaped.
Docker will return with
`container process caused: exec: "''": executable file not found in $PATH`
* feat(detectExecuteScan): generate ip result json
json will currently only be created in success cases.
No information about policy violation details available in the step yet.
* update report name
* Update cmd/detectExecuteScan.go
Co-authored-by: Giridhar Shenoy <giridhar.shenoy@sap.com>
* Update cmd/detectExecuteScan.go
Co-authored-by: Giridhar Shenoy <giridhar.shenoy@sap.com>
* Update cmd/detectExecuteScan_test.go
Co-authored-by: Giridhar Shenoy <giridhar.shenoy@sap.com>
* Update cmd/detectExecuteScan_test.go
Co-authored-by: Giridhar Shenoy <giridhar.shenoy@sap.com>
* move blackduck api package
* fix broken unit test
* added tests for new api calls
Co-authored-by: OliverNocon <oliver.nocon@sap.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
it is possible to overwrite the entrypoint for docker execution:
https://docs.docker.com/engine/reference/run/#entrypoint-default-command-to-execute-at-runtime
This is ideally done by passing `entrypoint=''` and not pass two options to the call.
This also helps with escaping issues of the empty value on other systems.
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Added checkIfStepActive step
* Implemented npmScripts condition. Code was refactored
* Added some unit tests
* Fixed go modules
* Fixed go modules
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Improved logging of splunk connectivity errors
* Splunk logging
* Moved error logging message
* Bugfix for response body
* Moves response body check, logging of connectivity errors
* Reformatting
* Adds check if response body is nil
* Toolrecord framework -
provide a common entry point for post processing code scan results
Changes to be committed:
new file: pkg/toolrecord/REAMDE_toolrecord.md
new file: pkg/toolrecord/toolrecord_main.go
new file: pkg/toolrecord/toolrecord_test.go
* Add toolrecord file to Checkmarx results
modified: cmd/checkmarxExecuteScan.go
* Add toolrecord file to Fortify results
modified: cmd/fortifyExecuteScan.go
* Add toolrecord file to Whitesource results
modified: cmd/whitesourceExecuteScan.go
* unset umask (#2927)
* (feat) adds error logging output for downloading reports from whitesource (#2928)
* Add toolrecord file to Protecode results
* address code climate findings (1/2)
* address codeclimate findings (2/2)
* add comments to all methods
* Toolrecord library:
- move all toolrun files into a subdirectory
- fix timestamp generation in filenames
* add protecode group's URL to toolrecord data
* fix syntax error from previous commit in cmd/protecodeExecuteScan.go
* toolrecord: fix projectVersionID and generated URLs in fortifyExecuteScan.go
* cmd/fortifyExecuteScan.go: replace a hard-coded servername with
config.ServerURL
* update description
* add toolrecord file to detectExecuteScan
* toolrecord/whitesource: add project names as context
Co-authored-by: Kevin Stiehl <kevin.stiehl@numericas.de>
Co-authored-by: ffeldmann <felix@bnbit.de>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* using default java truststore
* default java keytore
* remove trust store
* working directory
* change dir java_home
* env variable
* trying to find jre home
* changing directory to jre home and java home
* trying java jre
* remving $
* trying to search the cacerts
* copying existing cacerts
* removing change directory
* searching for secrutiy folder only
* searching cacerts
* new path for cacert
* path to ca-cert
* new trust store
* changing cacert location
* only adding maven_opts env variable once
* log message
* ca cert path from user
* handelling interface modification
* enhance logs and code clean up
* code clean up
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* feat(config):read config/defaults with authentication
This change allows to use defaults and config files from a protected GitHub repository.
The options `--customConfig` and `--defaultConfig` already allowed to provide a link to an uprotected file.
Now, by passing a value in the form `<hostname>:<token>` to parameter `gitHubTokens` (this parameter can be passed multiple times) a token can be provided for dedicated hosts.
This makes it possible to use a link like
`https://api.github.com/repos/SAP/jenkins-library/contents/resources/my-defaults.yml?ref=master`
as reference to a default file or similarly as reference to a configuration file.
* update generation to allow protected config/defaults
* fix CodeClimate issues
* update missing generations
* new step integrationArtifactTriggerIntegrationTest
* add new step into allow list
* add the new step to main command
* refer cpe
* remove unused unit tests
* Check methods and URLs of http request
* Add TriggerIntegration to mockingutils
* Format code
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Linda Siebert <linda.siebert@sap.com>
* Switch to service key for CPI GetMplStatus
Introduces read method for service key files, mock utils and tests.
* Use secret text instead of file
* Change serviceKey definition
* Update cpiUpload to use Service Key
retrieved the host and uaa information from service key
* Update cpiDeploy to use service key
retrieved the host and uaa information from service key
* Update cpiServiceEndpoint to use Service Key
retrieved the host and uaa information from service key
* Update cpiDownload to use Service Key
retrieved the host and uaa information from service key
* Update cpiUpdateConfig to use Service Key
retrieved the host and uaa information from service key
* Refactor serviceKey var name
* Fixed references to service key to follow the real format
they should be accessed through oauth instead of uaa because of the format of the json
* Rename ServiceKey to APIServiceKey
To support having a different service key(and for readability), we need to change the name to API.
* Add STAGES and STEPS yaml
add in to each yaml file of cpi integration
* Revert "Add STAGES and STEPS yaml"
This reverts commit aa2665d158.
* Change comments/formatting commonUtils
Make comments more understandable and follow code climate suggestions
* Change documentation files for steps
remove OAuth and host and change credentials to be servicekey
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
* make use of new read,writePipelineEnv Steps in groovy
* remove unused cat
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Make sonarExecuteScan orchestrator-agnostic
* Increase coverage + support empty or false env vars
* Use cleared env for unit tests
* Refactor to standalone package
* Fix review findings
* Fix review findings
* Fix unit test
* Add logging
* Refactor
* Add to codeowners 😎
* Apply suggestions from code review
* Remove unreachable code
* no message
* fix typos
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* modifying detect.maven.excluded.scopes from TEST to test
* new maven alt deployment flags
* changing flag names
* tlsCertificate addtion
* adding publish flags
* new flags
* publish flag
* enhance maven builds
* enhance maven builds
* creating new settings xml
* updating project settings
* changing interface for artifactPreparation that uses the same maven util niterface
* adding general scope to maven params
* global reference
* removing vault tmp
* debuging deployment user
* more debug
* maven build paras
* using smaller case
* adding incorrect error check
* adding deployment flags
* code refactor
* unit tests
* changing scope of paramter for tls certs
* new scope for tls
* remove trailing space in mavenBuild.yaml
* trailing space fix
* typo fix and jenkins secret
* including jenkins credentials for repo pass in the maven build groovy
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Adds headCommitId, which stores the head commit has of the current build, includes tests and generated files
* Adds headCommitId, which stores the head commit has of the current build, includes tests and generated files
* Go fmt fix
* Fixes artifactoryPrepareVersion test
* Removes xMake CommitId
* "ignoreSourceFiles" parameter removed from general section and added to specific build tool section
* Update configHelper_test.go
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Don't fail if components list is empty. Resolves failures when scanning images from Crossplane.
* Update formatting with go fmt
* Update pkg/protecode/protecode.go
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Update pkg/protecode/protecode.go
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Fix change, make consistent
Co-authored-by: d.small@sap.com <d.small@sap.com>
Co-authored-by: dee0 <dsmallzero@gmail.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* add versioningModel parameter
* extract versioning model to own package
* move log message
* use versioning method
* add customScanVersion parameter
* use customScanVersion
* adjust docs on other steps
* update test case
* Add dir to whitesource scan
* Add default for "dir" option
* Change param name to workDir
* Change param name WorkDir to ScanPath
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Fix project lookup
* Added test for space
* Update pkg/fortify/fortify.go
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* added support for test credentials
Co-authored-by: Kevin Stiehl <kevin.stiehl@numericas.de>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Introduce transport request create solman
* log the transport request id also in case creation failed
maybe there are some rare cases where a transport request id is returned nevertheless.
* report exit code always
* inline parameter
* Provide more parameters in log message, might help with troubleshooting
Co-authored-by: Roland Stengel <r.stengel@sap.com>
* Unit Test Assemble Package
* Remove obsolete lines
dust wiping
* climate change
* climate change #2
* climate change #3
* climate change #4
* climate change #5
* NSPC serial builds
* Actual Delivery Commit
* Download Delivery_logs.zip
* Publish Result
* Testing
* !Polling
* Provide Commit to BF only if set
* dust wiping
* More Dust to Wipe
* Publish more than on file
* Write Log for Publish
* fix unit test (now Dummy Entries)
* save one line of code for climate change
* Update cmd/abapEnvironmentAssemblePackages.go
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* More Detailed Log Messages
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* update data type of influx measurements
* Update checkmarx.yaml
* pick changes from #1885 for testing
* update generated code
* update to new datatype
* adjust to type changes
* change back to string type
* Update fortifyExecuteScan.go
* add typo to be backward compatible
* change type to int for files_scanned and lines_of_code_scanned
* add typo
* add measurements to whitesource
* update generated sources
* adjust test cases
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Add sca cmd extensibility
* Fix formatting
* HTTP retry
* Improve handling of retry on timeout
* Go fmt
* Fix test
* Fix test
* Test stability
* Fix test
* Fix test
* Fix test
* Update fortifyExecuteScan.go
do not fail step in case mta contains only node modules.
No pom.xml required then.
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Go Unit Tests fail due to windows/linux
file separator mismatch. See issue 2660.
* review results
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Improvements
* Formatting
* Fix test
* Update resources/metadata/fortify.yaml
Enhance description
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Unify version handling with ws step
* Part 2
* go fmt
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* fix(whitesource): remove parameters from scan call
parameters are forced into the config thus they don't need to be passed again.
* add proper project to config
* fix(whitesourceExecuteScan) correctly handle structured projects
report file names of projects which had a structure in the name
like `@test/myProject ` were incorrectly handled
This now prevents that reports are targeted to a sub directory.
Structure is now part of the filename.
* fix CodeClimate finding
* Provide getters for stdout, stderr on ShellRunner, ExecRunner
we need that in order to set the streams back in case we have to scan the command
output ourselvs during some function calls.
In case of policy violations, the scan is failed and default behavior is to not update the project in the WS system.
See docs at https://whitesource.atlassian.net/wiki/spaces/WD/pages/33914890/Maven+Plugin.
Since this is inconvenient and we also changed the behavior already for NPM, we force the update.
* LogRange provide git log ref1..ref2
This we need for checking inside the commit range
for transportRequestIds and changeDocumentIDs in the
body of the commit message.
* LogRange provide git log ref1..ref2
This we need for checking inside the commit range
for transportRequestIds and changeDocumentIDs in the
body of the commit message.
* [refactoring] Make room for other upload action: move package
In the near future we will have more upload actions, like
SOLMAN, RFC. Here we prepare the package structure for that.
* don't use aliasing
* rename entities (no leading CTS)
From the current location inside "cmd" the npmExecutorMock cannot
be used from any coding inside "pkg".
When we would like to re-use the npm functionality we have also to
provide tests and this requires having a mock.
In order to be able to use the mock from pkg we move the mock from
"cmd" to "pkg" into package "npm".
With the shift from package "cmd" to "npm" a lot of fields in the mock
has been made public.
Up to now we get a message 'Requested resource could not be found' which is not very
helpul during troubleshooting based on the log. With this change we tell the reader
which resource could not be resolved.
First a bug fix is addressed in which the pull policy could not be configured to false by configuring the general configuration. It could neither be configured via dockerExecute or dockerExecuteOnKubernetes, even though this parameter is docker specific. Only by configuring the specific step where one wants to set the pull policy to false can it be configured.
As the bug stems from zero values being in the context config map, which is also addressed with this PR. That is the second part: Context config parameters are only set if they have a value.
* Don't set pull image if not configured
Otherwise, if the pull policy is not set explicitly for a step, dockerPullImage is set to true. Thus, before this change, it cannot be set in the general, or in dockerExecute or in dockerExecuteOnKubernetes configuration.
* Fix unit tests
* Add pullImage parameter test
* Do not place empty default values in context config
* Use putIfNotEmpty for sidecar container options
* Export common configuration
Keys that are set by both main and sidecar container can be exported
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Upload in Chunks
* fix unit tests for file upload in chunks
* upload in chunks review round 1
* Upload in Chunks - review round 2 - comments
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
Test verifies that correct pom.xml is used as provided in the parameters
Also removed the --file pom.xml in the tests as maven should take pom.xml as default
Up to now the code generator is not able to handle the type
map[string]interface{} which is important for nested
configurations.
With that change we support such nested configuration.
Fo now parameters with a map type are not supported via
command line parameters. Those parameters are simply
ommitted. But with this change is it possible to read
such nested structures from the pipeline configuration
(.pipeline/config.yml).
As a next step we can discuss if we would like to support
such values also via command line parameters. One possible
approach could be
```
./piper <command> -myParam key1=val1 --myParam key2=val2
```
which gets finally collected inside our map:
```
map["key1"] = "val1"
map["key2"] = "val2"
```
This is of course hard to do for deeper nestings. In that case
providing a pointer to a file might be more suitable.
In that context we need to consider how to
- declare the default values for map like parameters in our
metadata files.
- deal with the different types we have for the parameter
itself wrt the yaml like config on the one hand and on the
level of the command line parameters on the other hand. Maybe for
that we have to extend the metadata format (e.g. describe an
alternate type receiving the values from the command line, like
[]string. With that approach values for simple nested (... not deep
nested) params can be provided like described above, it would be
possible to represent these parameters for the command line parser
as string slice entries like "[]string{key1=val1, key2=val2". These
parameters needs in this case transformed "by us" into the map we
use further down the road.
In case we agree in principle on an approach as outlined here we should
adjust the golden files reflecting this use case.
* initial commit of yaml file
* initial commit for HaDoLint in GO
* add helper function to load file from url
* load config file
* write report information to disk
* comment the code
* refactor groovy code
* remove download function from FileUtils
* use http.Downloader
* rename step files
* update generated files
* update generated files
* remove duplicate commands
* add credentials for config url
* add generated test file
* reuse piperExecuteBin functions
* correct step name
* update go step
* deactivate test
* fix import
* use differing go step name
* rename step
* correct result publishing
* correct command name
* expose tls insecure flag
* hand through error
* disable tls verification
* fix tls disabling
* use credentials
* mow
* reformat
* add qgate only if set
* correct report name
* remove old defaults
* add qgate to defaults
* handle report name
* restore default
* remove unused step config
* use piperExecuteBin
* remove obsolete type
* add test cases
* remove groovy tests
* move client parameter handling to run function
* use custom interfaces and mockery
* remove commented code
* correct struct names
* rename parameter dockerfile
* add further asserts
* cleanup
* change file permission to read/write
* remove tokenize
* add further comments
* init http client only if necessary
* add todo
* Revert "rename parameter dockerfile"
This reverts commit 2a570685b8.
* add alias for dockerfile parameter
* correct test case
* Apply suggestions from code review
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add comment about mock assertions
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Is there any benefit from having
```
assert.Error(./.)
assert.EqualError(./.)
```
?
assert.Error ensures that we have an error.
assert.EqualError ensures that we have an error and
moreover it checks for a specific error. Hence
assert.EqualError does all and more what assert.Error
does.
In case there is a benefit from that pattern this PR should not be merged.
In case there is not benefit from that pattern we should abandong that pattern.
* Make sure the UA scan is known to the scan object. Fixes downloading reports later on.
* Move polling into pkg/whitesource, add test for e2e scan
* Remove conditions from stash config resource
* Don't use version stored in CPE. This will prevent the versioningModel from being applied.
This change extends the npmExecuteScripts step to support execution of
npm scripts for specific modules. Previously, it was not possible to
execute npm scripts only for specific modules. Now, if the parameter
buildDesriptorList is set the scripts defined by the runScripts
parameter will be executed for the modules defined by
buildDescriptorList. Note, in this case the buildDescriptorExcludeList
will be ignored.
* makes containerImage not mandatory
* Adds kubectl container
* Adds log statement to debug
* adds general to container image
* removes GENERAL again
Removes condition from Kubectl container
* removes workDir
* marks logs as debug
* adds workingdir again
* Adds author to commits
* Adds commit time now
* remove deprecated and reorder
* adds deprecated again to containerRegistryUrl
Adds GENERAL scope to containerImage
* updates generated file
* Renames containerImageNameTag
* adds else case
* adds debug log
* code cleanup
* adds debug log
* revert
* adds debug logs
* revert
* makes root path not hidden
* revert
* Read container properties
* Removes debug message
* Removes debug message
* Removes general scope again
* Fixes unit test
* Adds helm capabilities to the gitopsUpdateDeployment step
* Adds helm capabilities to gitopsUpdateDeployment step
* Removes condition from input field
* Adds test for invalid deploy tool
* Fixes typo
* Adds tests for git errors and file errors
Simplifies test setup
* Adds test for error on image name extraction
* fixes URL variable name
* adds workind directory to paths
* Refactors too long method
* Reverts refactoring method
* Adds repository name as parameter
* Adds glob method
* Test glob method
* Revert "Test glob method"
This reverts commit ac11b54c14.
* Revert "Adds glob method"
This reverts commit ddf47ddebe.
* Revert "Adds repository name as parameter"
This reverts commit 8fc471c909.
* Removes getWd
* Adds stash deployDescriptor
* removes = from paramters
* Revert "removes = from paramters"
This reverts commit 3ecb3665e2.
* Adds " around parameters
* adds logging of all files
* Updates helm to version 3.3.4
* Clean up debug logs
* Raise error if no branch name provided.
Defaulting should be handled by step configuration.
* clean code
* Updates fields and adds checks for required field for certain deploy tools
* Fixes default commit message
* Update long description
* Removes default parameter
* Update resources/metadata/gitopsUpdateDeployment.yaml
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Updates yaml file
* Add error category and removes too much wrapping
* Update generated file
* Checks all parameters before returning the error
* Introduces constant
* Renames constant
* Fixes unit tests
* unexpose constants
* Makes tests thread safe and resilient to failed deletion
* Remove methods that did not work properly with hash containers rather than tags.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Should avoid issues with this file being owned by root (perhaps via running in docker container), preventing the workspace from being cleaned properly.
* makes containerImage not mandatory
* Adds kubectl container
* Adds log statement to debug
* adds general to container image
* removes GENERAL again
Removes condition from Kubectl container
* removes workDir
* marks logs as debug
* adds workingdir again
* Adds author to commits
* Adds commit time now
* remove deprecated and reorder
* adds deprecated again to containerRegistryUrl
Adds GENERAL scope to containerImage
* updates generated file
* Renames containerImageNameTag
* adds else case
* adds debug log
* code cleanup
* adds debug log
* revert
* adds debug logs
* revert
* makes root path not hidden
* revert
* Read container properties
* Removes debug message
* Removes debug message
* Removes general scope again
* Fixes unit test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* add vaultSecretFileReferences
* fix test
* fix test
* go generate
* remove code duplication
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* Adds piper step to update deployment configuration in external git repository.
https://github.wdf.sap.corp/ContinuousDelivery/piper-ita/issues/21
* Adds handling of branchName as an optional parameter
* Update resources/metadata/gitopsUpdateDeployment.yaml
Feedback about description
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Adapt to interface guide
* Refactors to GitopsExecRunner
* Refactors to GitopsExecRunner in test
* Removes unnecessary mocked methods
* Adds tests for git utils
* Adds new step to CommonStepsTest.groovy
* Updates description from yaml
* Restricts visibility of methods and interfaces
Adds comments where necessary
* Updates comments
* Fixes URL name
* updates description
* updates generated file
* Fixes compile issue in CommonStepsTest.groovy
* Updates long description
* Updates test to run green on all kind of OS
* Removes global variables from tests
* Default branch: master
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Typo in Hierarchy
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Refactors test to allow parallel execution
* Renames utility variable in gitopsUpdateDeployment.go
* Renames error variables in gitopsUpdateDeployment.go
* simplified parameters for kubectl
* Refactors util classes to use parameters rather than global variables
* makes username and password mandatory
* remove unnecessary mandatory flag
* remove new methods from mock that are not necessary
* replaces with EqualError
* replaces with NoError
* update generated file
* refactor tests
* refactor tests
* make tests parallel executable
* parallel execution of tests
* Refactors interfaces to stop exposing interfaces
* Feedback from PR
* Simplifies failing mocks
* Renames variables and interfaces
* Fixes error messages
* shorten variable names
* Renames unused parameters in tests
* Cleanup nil parameters
* Typo
* Wrap errors and remove unnecessary logs
* Remove containername and filePath from GENERAL scope
* correct generated file
* corrects expected error messages
Co-authored-by: OliverNocon <oliver.nocon@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* remove docs generator code from step-generator
* add docs generator to dedicated package
* add test cases
* add entry point for docs generation
* make output more readable
* read additional defaults
* add custo defaults parameters
* remove commented code
* adjust custom default parameter in workflow
* remove conflict leftovers
* handle custom default values
* remove comment
* extract code to function
* extract metadata reading to function
* do not print empty strings in favor of PIPER_* env vars
* extract new code to own metadata file
* only reset default on booleans
* remove obsolete test case
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* add type to sonar field
* respect type of influx fields
* update generated code
* switch type
* copy changes from #1885
* log JSON data
* read simple values from json
* Update InfluxData.groovy
* Revert "Update InfluxData.groovy"
This reverts commit c8cfdf381f.
* Revert "read simple values from json"
This reverts commit 94b69866d2.
* Revert "copy changes from #1885"
This reverts commit 2471b4475e.
* update TODO
* handle non-string values as JSON
* change value type to interface in resources
* regenerate code
* add test cases
* handle reading of json files
* write json data to json files
* fix assignment
* use GetResourceParameter
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add code block type
* add parameters to hand in library and binary name
* use library and binary name parameters
* add test cases
* use yaml file to distintuish custom from regular steps
* add test case
Vars file handling centralized
We have the same coding for handling varsf-files and vars. With that change we shift to having one common coding for that
* remove docs generator code from step-generator
* add docs generator to dedicated package
* add test cases
* add entry point for docs generation
* make output more readable
* remove dead code
* fix redundant type issues
* cleanup
* extract report function for protecode package
* use speaking status constant for API results
* remove unconsidered return value
* correct switch statement
* handle severe vulnerabilities
* Apply suggestions from code review
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* correct test name
* return errors from WriteReport function
* expose ReportData struct
* set Error Category
* refactor constant visibility
* change type name
* describe type
* change type name
* fail after report generation
* do not fail on report write errors
* add error as return value
* fix typo
* use require statements
* assert major vulnerabilities
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Fix
* Adapt errors
* Consider unexpected JSON
* defer closing the response body
* Add comments to explain function
* Improve assert statements semantically
* Change comment format due to CodeClimate
* Extract sub function
The SAP NPM registry has been migrated to the default public registry,
thus the separate configuration with the sapNpmRegistry is not required
anymore.
All packages from npm.sap.com have been migrated to npmjs.org
and in the future SAP packages will only be available from the default
public registry.
Currently, the mtaBuild step installs the wrong artifact in a spring project making use of the "repackage" feature. This PR fixes that by checking if an ".original" jar file exists and using that instead.
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Adapt changes of abautils pkg
* Add test for polling
* Minor fix
* Fix yaml spacing
* Add longdescription to yaml
* Refactor abaputil methods
* Refactoring
* Refactoring
* Minor fix
* Minor fixeds
* Adapt to new abaputils.AUtilsMock
* Delete obsolete initial checks for params
* Fix manageGitRepoUtils_test.go
* Adjust pollEntity tests
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
This change adds a buildDescriptorExcludeList parameter to
npmExecuteScripts, to enable the exclusion of certain directories when
executing npm scripts. Previously, npmExecuteScripts could only execute
scripts in all packages.
Now it is possible to provide paths or patterns as elements of the
buildDescriptorExcludeList to exclude packages when executing npm scripts.
* Dont work upon a global command.Command instance inside cloudfoundry package
o Up to now we work on a private and shared instance of command.Command inside
the cloudfounrdy package. We need to be able either configure this instance
(environment variables) according to the use case. One option is to hand over
an already configured instance which is used elsewhere. This is what we do with
this commit.
o With this commit we remove the instance which is shared within the cloudfounrdy
package and to provide an instance with a receiver which gets handed over to the
functions. Hence we are thread save: parallel invoctation of e.g. Login will not
affect each other.
o Up to now we work on a private and shared instance of command.Command inside
the cloudfounrdy package. We need to be able either configure this instance
(environment variables) according to the use case. One option is to hand over
an already configured instance which is used elsewhere. This is what we do with
this commit.
o With this commit we remove the instance which is shared within the cloudfounrdy
package and to provide an instance with a receiver which gets handed over to the
functions. Hence we are thread save: parallel invoctation of e.g. Login will not
affect each other.
* Added Vault package
* added support for logical path lookups instead of api paths
* added integration tests
* add integration tests and mock tests
* Replace mock with mockery generated one
* update tests to use mockery
* create mocks sub package
- mock is used now for cf api commands
- tests for cf api and cfk login are checking which
commands are really executed
- some minor simplifications wrt asserts
* [refactoring] move the shell/command related interfaces into pkg/command
otherwise we are not able to use the corresponding mocks for the items contained in pkg since
these interfaces are not visible from the pkg folder
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Whitesource MVP for Gradle, Golang, and NPM/Yarn
* Refactoring
* Refactor and cleanup, better error checking
* publish stepResults, use pkg/versioning, bubble up errors, add gomod versioning support
* Run gofmt and cleanup comments
* Resolve PR comments
* Update resources/metadata/whitesource.yaml
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Only determine project coordinates if they are missing
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Gradle versioning artifact
* fix gradle artifact version regexp and refactor
* Fix token extraction from output buffer
* Fix some issues with pip and jsonfile versioning logic
* Remove useless spacing
* Remove unnecessary test file and fix naming style for JSONDescriptor
* Automatically download wss-unified-agent if file does not exist
* adds downloadVulnerabilityReport, checkSecurityViolations, minor refactoring
* adds config.ReportDirectoryName, improves readability
* Version-wide reporting for vulnerabilities and list of libraries.
* Refactor and improve build accuracy
* fix sed command
* Add includes file pattern config option
* Adds --exclude command line flag
* run go mod tidy and regenerate step framework
* Fix unit tests
* revert changes
* poll project status before downloading reports
* merge with master
* go mod tidy, go fmt, and fix whitesource unit test
* sync go.mod
* sync go.mod again
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Allow retrieving exit code from command execution
This will be helpful to derive error categories in case
an executable provides context-specific error codes.
* make sure that we always have a non 0 exit code for errors
