* lower tls related log messages from info to debug level
* remove protcodeExecuteScan related warnings
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* changes to detectExec before master merge
* changes for detectExecuteScan
* self generated code added
* fix syntax errors and update docu
* added unit tests for fail and Group
* fix failOn bug
* add Groups as string array
* add Groups as string array
* tests and validation for groups, failOn
* Updated docs and added more tests
* documentation md files should not be changed
* Handle merge conflicts from PR 1845
* fix merge errors
* remove duplicate groups, merge error
* adding buildCode and buildTool as params
* switching build options
* building maven modules
* parameter correction
* parameter correction
* gnerate with new build parameter
* adding comments
* removing piper lib master and modifying goUtils to download 1.5.7 release
* first cleaning then installing
* multi module maven built
* multi module maven built removing unwanted code
* multi module maven built moving inside switch
* testing
* modifying the default use case to also call maven build
* modifying the default use case to also call maven build wih --
* corrected maven build command
* corrected maven build command with %v
* skipping test runs
* testing for MTA project with single pom
* adding absolute path to m2 path
* clean up
* adding switch for mta and maven and removing env from containers
* commiting changes for new detect step
* correting log message
* code clean up
* unit tests changes to detectExecute
* basic tests for new change
* restoring piperGoUtils to download correct piper binary
* code clean up
* code clean up
* protecodeExecuteScan -> Added authentication with user API key
* protecodeExecuteScan -> updating .yml file
* protecodeExecuteScan -> go generate fixed
* protecodeExecuteScan -> naming convention applied for UserAPIKey parameter
* protecodeExecuteScan -> extending groovy code for mapping jenkins credentials
Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Initially generated tmsUpload<...> files
* First provisioning of parameters supported by tmsUpload step
* Refer to Go step from tmsUpload.groovy
* Initial client implementation
* Reverting line delimiters in tmsUpoad.groovy back to Unix ones
* Temporarily remove when-condition for Release stage
* Define useGoStep parameter in tmsUpload.groovy
* Unstash buildResult if useGoStep is true
* No unstashing and empty credentials, when using go step
* Register TmsUploadCommand in piper.go
* Cleanup groovy-related changes - they will be temporarily implemented in a different repo
* Make getting OAuth token success
* Look through the code and cleanup it a bit
* Read service key from Jenkins credentials store
* Provide initial set of unit tests for methods in /pkg/tms/tms.go file
* Minor improvements on logging response on http call error
* Check, if positive HTTP status code is as expected
* Cleanup tms.yaml file, provide additional unit test for tms.go
* Provide unit test for the case, when request body contains spaces
* Specify nodeExtDescriptorMapping parameter as of type map in tms.yaml
* Implement client method for getting nodes
* Write tests for GetNodes method
* Add GetMtaExtDescriptor client method and cover it with unit tests
* Provide first implementation for Update- and UploadMtaExtDescriptor
client methods
* Provide first implementation for Update- and UploadMtaExtDescriptor
client methods
* Provide UploadFile and UploadFileToNode client methods
* Provide tests for Update- and UploadMtaExtDescriptor client methods
* Write tests for FileUpload and FileUploadToNode client methods
* Minor corrections
* Remove some TODO comments
* Rename some of response structures
* Revert change for line delimiters in cmd/piper.go
* Add uploadType string parameter to UploadFile and UploadRequest methods
of uploader mock to reflect the changed Uploader implementation
* Start to implement execution logic in tmsUpload.go
* Changes in tms.yaml file
- remove resources from inputs in tms.yaml and implement mtaPath
parameter settings in the yaml file the same way, as it is done in
cloudFoundryDeploy.yaml
- rename tms.yaml to tmsUpload.yaml, since some generation policy
changed meanwhile
* Rename tms.yaml to tmsUpload.yaml and do go generate
* Use provided proxy on communication with UAA and TMS
* Set proxy even before getting OAuth token
* Further implementation of tmsUpload.go
* Continuation on implementing the tmsUpload.go executor
* Get mtarFilePath and git commitId from commonPipelineEnvironment, if
they are missing in configuration file + minor changes
* Implement a happy path test for tmsUpload logic
* Cover with unit tests main happy and error paths of tmsUpload.go logic
* Extend set of unit tests for tmsUpload.go
* Eliminate some TODOs, extend unit tests for tmsUpload.go
* Delete some TODOs
* Remove a couple of more TODOs from tms_test.go file
* Provide additional unit test for error due unexpected positive http
status code on upload
* Revert back line delimiters in cmd/piper.go
* Comment out file uploading calls in tmsUpload.go
* Run go generate to update generated files
* Convert line delimiters in tmsUpload.yaml to Unix ones, as well as
provide new line character in the end of the file, plus minor fix for
logging in tmsUpload.go file (pipeline complained)
* Correct description of a parameter in tmsUpload.yaml, extend unit tests
to check for trimming a slash in the end of TMS url for client methods
that do upload
* [minor] Add a comment in the test code
* Add stashContent parameter to do unstashing in tmsUpload.groovy, remove
some of the clarified TODOs
* Uncomment uploading file calls in tmsUpload.go, declare buildResult
stash in tmsUpload.yaml
* Remove clarified TODOs from the tmsUpload.go file
* Run go fmt for jenkins-library/pkg/tms
* Do not get explicitly values from common pipeline environment - all
configurations are provided in yaml file
* Remove unused struct from tmsUpload_test.go
* Run go fmt jenkins-library\pkg\tms
* Revise descriptions of parameters provided in tmsUpload.yaml file
* Specify STAGES scope for tmsUpload parameters
* Provide STAGES scope for the tmsUpload parameters, provide default value
for stashContent parameter
* Remove trailing space from tmsUpload.yaml
* Provide unit tests for proxy-related changes in http.go file
* Improve proxy implementation in tmsUpload.go file
* Make tmsServiceKey again a mandatory parameter
* Run go generate command to make the generated files correspond the yaml
state
* Change line delimiters back to Unix ones (were switched while resolving
the conflicts)
* Remove trailing spaces from tmsUpload.yaml
* Minor change in a comment to trigger pipelines with commit
* Improve checks for zero-structs and for empty maps, as well as use
different package to read files in the tests
* Revert line endings in http.go
* Revert comments formatting changes in files that do not belong to the tmsUpload step
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Reorders getApiInformation, changes variables to get start time, adjusts and adds test cases
* Changes the way to get apiInformation and reduces number of requests
* Changes getting pipeline start time from correct env variable
* Refactors getApiInformation functionality
* Adds GetBuildReason() for Azure and Jenkins
* Updates JobURL for ADO
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetLog() function to orchestrator
* Fixes BUILD_NUMBER env variable
* Fixes correct env var for JENKINS_HOME
* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation
* Adds possibility to read errorJsons; updates splunk package for log files (WIP)
* Uncommenting dev code
* Adds GetRequest function which holds the response in memory (not saved to disk)
* Implements GetLog() function for ADO, adds function to read PipelineRuntime
* PAT has been revoked
* Changes http package, s.t. if password only is required basic auth works too
* Adds env variable for azure token, error handling in case of unauthenticated/nil response
* Adds logging output in case env variable can not be read and fallback variable needs to be used
* Adds usage of environment variables for auth, uses jenkins api
* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function
* Adds initaliziation function for orchestrator authetnication
* Adds settings struct for orchestrator authentication
* Adds function to whole logfile to Splunk
* Struct for pipeline related telemetry information
* Increase messagebatch size to 10k
* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk
* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation
* Changes content type to txt
* Send telemetry independent of logfiles, increases amount of messages per file
* Adds JobURL for orchestrators and UnknownOrchestrator as fallback
* telemetry makes use of orchestrator specific information
* Adds orchestrator independent correlationID
* Adds custom fields for pipeline status
* go fmt
* Removes env var test - no env variables are read anymore
* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized
* Removes Custom fields from telemetry as these can not be reflected in SWA
* Adds custom telemetry information (piperHash,..) to each step telemetry information
* Removes falltrough in case no orchestrator has been found
* Updates tests for orchestrator package
* Adds orchestrator import in generated files
* Updates generator files for internal library
* Adds orchestrator telemetry information to steps
* Updates generated files, fatalHook writes to cpe
* Go generate from master, go fmt
* Adds Custom Data field LastErrorCode
* Removes GetLog() test
* Update init_unix.go
* Update docker_integration_test_executor.go
* Update integration_api_cli_test.go
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Reverts go1.17 fmt formatting
* Renames customTelemetryData to stepTelemetryData
* Adjustments to orchestrator-package, cleanup, adds JobName
* Adjusts commonPipelineEnvironment path
* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package
* Go fmt
* Changes path for errorDetails, adds debug information
* Removes custom fields from step, adds orchestrator, commithash to baseMetadata
* Adjusts tests for telemetry package
* Adds tests for orchestrator
* Updates generated files, initalization of splunk client only if its available in the config
* Fixes typo in helper go
* Update pkg/http/downloader.go
* Update pkg/http/downloader.go
* Update pkg/log/fatalHook.go
* Update fatalHook.go
* Update pkg/splunk/splunk.go
* Update pkg/telemetry/data.go
* Adds GetBuildStatus() and GetAPIInformation() to orchestrators
* error formatting
* Bugfix: dont send telemetry data if disabled, adjusts test
* go fmt
* Use correct error handling
* Update pkg/telemetry/telemetry.go
* Fixes telemetry disabled in the tests
* Fixes http tests
* Log fatal errors to logFile
* Adds CustomReportingConfig to hooks
* Cleanup comments in splunk package
* Adds possibility to send telemetry to custom endpoint
* Adds debug output for the payload
* Debug output for the payload as a string
* Adds test cases for changes in telemetry package
* go fmt
* Adds generated files for new step
* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17
* Adjusts test for sonarExecuteScan
* Adjusts test for sonarExecuteScan
* Adds explanation for customreportingConfig
* Makes disableing of customSend more obvious
* Adds custom step reporting to each step, updates generated files, adjusts helper testdata
* fixes unit test wrong usage of logging
* Send pipeline data altough there has been no error, adjust test cases
* Reverts changes for customReporting
* Updates generated files, removes customReporting
* Removes writing errorDetails to CPE
* Reverts usage of customreporting
* go fmt
* reverts changes in http_test
* reverts changes in http_test
* Skips integration cnb test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* changes to detectExec before master merge
* changes for detectExecuteScan
* self generated code added
* fix syntax errors and update docu
* added unit tests for fail and Group
* fix failOn bug
* add Groups as string array
* add Groups as string array
* tests and validation for groups, failOn
* Updated docs and added more tests
* documentation md files should not be changed
* Handle merge conflicts from PR 1845
* fix merge errors
* remove duplicate groups, merge error
* adding buildCode and buildTool as params
* switching build options
* building maven modules
* parameter correction
* parameter correction
* gnerate with new build parameter
* adding comments
* removing piper lib master and modifying goUtils to download 1.5.7 release
* first cleaning then installing
* multi module maven built
* multi module maven built removing unwanted code
* multi module maven built moving inside switch
* testing
* modifying the default use case to also call maven build
* modifying the default use case to also call maven build wih --
* corrected maven build command
* corrected maven build command with %v
* skipping test runs
* testing for MTA project with single pom
* adding absolute path to m2 path
* clean up
* adding switch for mta and maven and removing env from containers
* commiting changes for new detect step
* correting log message
* code clean up
* unit tests changes to detectExecute
* basic tests for new change
* restoring piperGoUtils to download correct piper binary
* code clean up
* code clean up
* protecodeExecuteScan :: fixing file upload for binaries
* protecodeExecuteScan :: fixing protecode generate file
* Fix upload test
* protecodeExecuteScan -> fixing tests
Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
* modify wrapper http to hold ca rot
* modifying the http client for maven build
* adding checks
* eliminating seperate jks
* test tls true
* insecure flag remove
* error debug
* storing cert
* sap root cert
* error
* only child cert
* test
* maven test
* moving outside loop
* changing pointer
* dont download existing certs
* typo fix
* removing mavenBuild test
* code clean up
* making hadolint using always cert true
* custom tls link for hadolint trust
* error handel
* extended condition for modifying custom tls
* unit test case
* checing when to add the customLinks
* not breaking existing hadolint client config
Co-authored-by: Your Name <you@example.com>
* Add sca cmd extensibility
* Fix formatting
* HTTP retry
* Improve handling of retry on timeout
* Go fmt
* Fix test
* Fix test
* Test stability
* Fix test
* Fix test
* Fix test
* Update fortifyExecuteScan.go
Up to now we get a message 'Requested resource could not be found' which is not very
helpul during troubleshooting based on the log. With this change we tell the reader
which resource could not be resolved.
The TransportTimeout value is now applied to the "response header
timeout" and "expect continue timeout" as is. Previously there
was a hard limit of 10 seconds and 1 second respectively (originating
from the article I based the previous PR on). While this doesn't
allow for fine-grained control, it is reasonable to apply the meaning
of "transport timeout" to any state or phase of a HTTP connection.
This change should solve the needs of some Piper clients to configure
very long response header timeouts of 5 minutes and above.
* Replace the default maximum request deadline with a default timeout on the transport level.
* Keep the possibility to set a maximum request deadline.
This might provide useful context to the user.
For example when you try to upload a SNAPSHOT jar to a maven-release
repo in nexus, this will print an error like
returned with response 400 Repository version policy: RELEASE does
not allow version: 1.0-SNAPSHOT
instead of
returned with HTTP Code 400
* add download file function
* add test case
* Update pkg/piperutils/FileUtils.go
* correct test case
* remove FileUtils.Download
* add Downloader
* add Downloader
* fix error
* respect header and cookies
* add test case
* rename files
* correct test case
* remove SendRequest
* correct test case
* Protecode as go implementation
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
