* added logic of fetching golang private packages for whitesource step and detectExecuteScan step
* changed logic of checking by config.PrivateModulesGitToken
* moved func prepareGolangPrivatePackages to golangBuild.go
* fix (gitOpsUpdateDeployment) add CA bundle options to plain clone and commit to trust enterprise github instances (#4602)
* downloading ca cert bundle when added as config
* adding logging statements
* allowing bats test to handle ca cert
* adding info message
* hard coding file names
* including correct http client util bundle
* removing logging message not needed
* adding cert bundle to commit and push
* improving the condition to add ca cert in commit and push
* fixing unit test
* fixing unit test
* fixing unit test
* fixing unit test
* fixing unit test
* feat(kanikoExecute): add dockerfilePath param to multipleImages (#4569)
* add containerDockerfilePath param to multipleImages
* rename ContainerDockerfilePath param to DockerfilePath
* Fix trailing spaces
---------
Co-authored-by: Egor Balakin <egor.balakin@sap.com>
Co-authored-by: Vyacheslav Starostin <32613074+vstarostin@users.noreply.github.com>
* fix(helm): forward sourceRepositoryCredentialsId from groovy to go layer (#4604)
forward sourceRepositoryCredentialsId from groovy to go layer in the same way how this is done for the targetRepositoryCredentialsId
* feat(config): exporting generateConfig function and applying minor changes (#4605)
* exporting generateConfig function and applying minor changes
* Added setConfigOptions to set configOptions variable.
Added possibility to set format output, json or yaml for now.
* Correcting mistake on cmd/getDefaults.go
Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>
---------
Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>
* moved func prepareGolangPrivatePackages to pkg/golang
---------
Co-authored-by: Akramdzhon Azamov <MY_NAME@example.com>
Co-authored-by: Andrei Kireev <andrei.kireev@sap.com>
Co-authored-by: Anil Keshav <anil.keshav@sap.com>
Co-authored-by: Egor Balakin <14162703+m1ron0xFF@users.noreply.github.com>
Co-authored-by: Egor Balakin <egor.balakin@sap.com>
Co-authored-by: Vyacheslav Starostin <32613074+vstarostin@users.noreply.github.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
Co-authored-by: Jk1484 <35270240+Jk1484@users.noreply.github.com>
Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>
For running open source vulnerability scans in de-coupled processes
it is helpful to allow that steps only create
compliance reports to inform users/teams
but not fail the pipeline.
This can now be achieved constitently with the flag:
`failOnSevereVulnerabilities`
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* Add ans implementation
* Remove todo comment
* Rename test function
Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
* Better wording
Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
* Add reading of response body function
* Use http pkg ReadResponseBody
* Check read error
* Better test case description
* Fix formatting
* Create own package for read response body
* Omit empty nested resource struct
* Separate Resource struct from Event struct
* Merge and unmarshall instead of only unmarshalling
* Improve status code error message
* Remove unchangeable event fields
* Separate event parts
* Change log level setter function
* Restructure ans send test
* Revert exporting readResponseBody function
Instead the code is duplicated in the xsuaa and ans package
* Add check correct ans setup request
* Add set options function for mocking
* Review fixes
* Correct function name
* Use strict unmarshalling
* Validate event
* Move functions
* Add documentation comments
* improve test
* Validate event
* Add logrus hook for ans
* Set defaults on new hook creation
* Fix log level on error
* Don't alter entry log level
* Set severity fatal on 'fatal error' log message
* Ensure that log entries don't affect each other
* Remove unnecessary correlationID
* Use file path instead of event template string
* Improve warning messages
* Add empty log message check
* Allow configuration from file and string
* Add sourceEventId to tags
* Change resourceType to Pipeline
* Use structured config approach
* Use new log level set function
* Check correct setup and return error
* Mock http requests
* Only send log level warning or higher
* Use new function name
* One-liner ifs
* Improve test name
* Fix tests
* Prevent double firing
* Reduce Fire test size
* Add error message to test
* Reduce newANSHook test size
* Further check error
* Rename to defaultEvent in hook struct
* Reduce ifs further
* Fix set error category test
The ansHook Fire test cannot run in parallel, as it would affect the
other tests that use the error category.
* Change function name to SetServiceKey
* Validate event
* Rename to eventTemplate in hook struct
* Move copy to event.go
* Fix function mix
* Remove unnecessary cleanup
* Remove parallel test
The translation fails now and again when parallel is on.
* Remove prefix test
* Remove unused copyEvent function
* Fix ifs
* Add docu comment
* Register ans hook from pkg
* register hook and setup event template seperately
* Exclusively read eventTemplate from environment
* setupEventTemplate tests
* adjust hook levels test
* sync tests- wlill still fail
* migrate TestANSHook_registerANSHook test
* fixes
* Introduce necessary parameters
* Setup hook test
* Use file instead
* Adapt helper for ans
* Generate go files
* Add ans config to general config
* Change generator
* Regenerate steps
* Allow hook config from user config
Merges with hook config from defaults
* Remove ans flags from root command
* Get environment variables
* Generate files
* Add test when calling merge twice
* Update generator
* Regenerate steps
* Check two location for ans service key env var
* Re-generate
* Fix if
* Generate files with fix
* Duplicate config struct
* Add type casting test for ans config
* Fix helper
* Fix format
* Fix type casting of config
* Revert "Allow hook config from user config"
This reverts commit 4864499a4c497998c9ffc3e157ef491be955e68e.
* Revert "Add test when calling merge twice"
This reverts commit b82320fd07b82f5a597c5071049d918bcf62de00.
* Add ans config tests
* Improve helper code
* Re-generate commands
* Fix helper unit tests
* Change to only one argument
* Fix helper tests
* Re-generate
* Revert piper and config changes
* Re-generate missing step
* Generate new steps
* [ANS] Add servicekey credential to environment (#3684)
* Add ANS credential
* Switch to hooks and remove comments
* Add subsection for ans
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
* Remove changes to piper.go
* Remove formatting
* Add test for ANS
* Define hook credential seperately from step credential
* Add test for retrieval from general section
* Add comment
* Get ans hook info from DefaultValueCache
* [ANS] Add documentation (#3704)
* Add ANS credential
* Switch to hooks and remove comments
* Add subsection for ans
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
* Remove changes to piper.go
* Remove formatting
* Add test for ANS
* Define hook credential seperately from step credential
* Add test for retrieval from general section
* Add comment
* Add documentation
* Review changes
* Review comments
* Improve documentation further
* Add note of two event templates
* Add log level destinction
* Further improvements
* Improve text
* Remove unused things
* Add ANS credential
* Switch to hooks and remove comments
* Add subsection for ans
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
* Remove changes to piper.go
* Remove formatting
* Add test for ANS
* Define hook credential seperately from step credential
* Add test for retrieval from general section
* Add comment
* Get ans hook info from DefaultValueCache
* Improvements
Co-authored-by: Linda Siebert <linda.siebert@sap.com>
Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
* New lines
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Roland Stengel <r.stengel@sap.com>
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
* adds return in gcs upload in case error occurs e.g. no credentials, avoid nil pointer dereference
* Adds generated files
* Updates generated files
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Add gcs upload to whitesourceExecuteScan step
* go generate
* patterns were updated
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* fixed generated output parameters for influx
* change name to lower case
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetRequest function which holds the response in memory (not saved to disk)
* Implements GetLog() function for ADO, adds function to read PipelineRuntime
* PAT has been revoked
* Changes http package, s.t. if password only is required basic auth works too
* Adds env variable for azure token, error handling in case of unauthenticated/nil response
* Adds logging output in case env variable can not be read and fallback variable needs to be used
* Adds usage of environment variables for auth, uses jenkins api
* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function
* Adds initaliziation function for orchestrator authetnication
* Adds settings struct for orchestrator authentication
* Adds function to whole logfile to Splunk
* Struct for pipeline related telemetry information
* Increase messagebatch size to 10k
* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk
* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation
* Changes content type to txt
* Send telemetry independent of logfiles, increases amount of messages per file
* Adds JobURL for orchestrators and UnknownOrchestrator as fallback
* telemetry makes use of orchestrator specific information
* Adds orchestrator independent correlationID
* Adds custom fields for pipeline status
* go fmt
* Removes env var test - no env variables are read anymore
* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized
* Removes Custom fields from telemetry as these can not be reflected in SWA
* Adds custom telemetry information (piperHash,..) to each step telemetry information
* Removes falltrough in case no orchestrator has been found
* Updates tests for orchestrator package
* Adds orchestrator import in generated files
* Updates generator files for internal library
* Adds orchestrator telemetry information to steps
* Updates generated files, fatalHook writes to cpe
* Go generate from master, go fmt
* Adds Custom Data field LastErrorCode
* Removes GetLog() test
* Update init_unix.go
* Update docker_integration_test_executor.go
* Update integration_api_cli_test.go
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Renames customTelemetryData to stepTelemetryData
* Adjustments to orchestrator-package, cleanup, adds JobName
* Adjusts commonPipelineEnvironment path
* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package
* Go fmt
* Changes path for errorDetails, adds debug information
* Removes custom fields from step, adds orchestrator, commithash to baseMetadata
* Adjusts tests for telemetry package
* Adds tests for orchestrator
* Updates generated files, initalization of splunk client only if its available in the config
* Fixes typo in helper go
* Update pkg/http/downloader.go
* Update pkg/http/downloader.go
* Update pkg/log/fatalHook.go
* Update fatalHook.go
* Update pkg/splunk/splunk.go
* Update pkg/telemetry/data.go
* Adds GetBuildStatus() and GetAPIInformation() to orchestrators
* error formatting
* Bugfix: dont send telemetry data if disabled, adjusts test
* go fmt
* Use correct error handling
* Update pkg/telemetry/telemetry.go
* Fixes telemetry disabled in the tests
* Fixes http tests
* Log fatal errors to logFile
* Adds CustomReportingConfig to hooks
* Cleanup comments in splunk package
* Adds possibility to send telemetry to custom endpoint
* Adds debug output for the payload
* Debug output for the payload as a string
* Adds test cases for changes in telemetry package
* go fmt
* Adds generated files for new step
* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17
* Adjusts test for sonarExecuteScan
* Adjusts test for sonarExecuteScan
* Adds explanation for customreportingConfig
* Makes disableing of customSend more obvious
* Adds custom step reporting to each step, updates generated files, adjusts helper testdata
* fixes unit test wrong usage of logging
* Send pipeline data altough there has been no error, adjust test cases
* Reverts changes for customReporting
* Updates generated files, removes customReporting
* Removes writing errorDetails to CPE
* Reverts usage of customreporting
* go fmt
* reverts changes in http_test
* reverts changes in http_test
* Skips integration cnb test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
image `mbtci` has been deprecated and is no longer maintained.
As per: https://hub.docker.com/r/devxci/mbtci
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
* Fixed validation for possibleValues option
* Change oneof-custom to possible-values
* go generate
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* feat: func for Docker config.json
Provide re-use function to create/update Docker conifg.json
* add comment
* update WhiteSource step
* fixes and additional tests
* Implemented validation for the option possibleValues
* Has been added the option mandatoryIf to config with validation
* Fixed issues found during code review
* improved golang template
* Fixed tests. Added validation for mandatoryIf option
* Fix typo
* Fixed tests
* Validation was refactored. Added options
* Added default value for parameters with possibleValues option
* Validation was moved after the configuration resolution
* Canceled some default values
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* feat(config):read config/defaults with authentication
This change allows to use defaults and config files from a protected GitHub repository.
The options `--customConfig` and `--defaultConfig` already allowed to provide a link to an uprotected file.
Now, by passing a value in the form `<hostname>:<token>` to parameter `gitHubTokens` (this parameter can be passed multiple times) a token can be provided for dedicated hosts.
This makes it possible to use a link like
`https://api.github.com/repos/SAP/jenkins-library/contents/resources/my-defaults.yml?ref=master`
as reference to a default file or similarly as reference to a configuration file.
* update generation to allow protected config/defaults
* fix CodeClimate issues
* update missing generations
* Add dir to whitesource scan
* Add default for "dir" option
* Change param name to workDir
* Change param name WorkDir to ScanPath
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
