1
0
mirror of https://github.com/SAP/jenkins-library.git synced 2024-12-14 11:03:09 +02:00
Commit Graph

162 Commits

Author SHA1 Message Date
gerstneralex
f5c33d51bb
Tms export (#4160)
* Change parameter type of nodeExtDescriptorMapping

(cherry picked from commit ca7ce0485a)

* Remove usage of the depricated ioutil package

(cherry picked from commit 9821915b33)

* Fix cmd failure if neither git/commitId nor customDescription are
provided

(cherry picked from commit c362681e45)

* Fix unit test

(cherry picked from commit 53a90aabb5)

* Step metadata, step code generation

* change type of nodeExtDescriptorMapping for export

* Refactoring and export implementation

* integration test

* Add export step

* Integration test

* format

* discard piper.go

* Review related changes

* restore piper.go

* remove unused method

* Extend documentation

* Add parameter useGoStep to tmsUpload.groovy

* Regenerate steps

* Rename function

* refactor constants

* Add error path tests

* Move some code to tms package

* Move more code to tms

* Combine tmsUpload, tmsUtils

* Add groovy wrapper

* add parameters to groovy step

* add import

* jenkinsUtils instance

* comment namedUser logic in groovy

* namedUser param

* remove logic for namedUser param

* Remove TMS integration tests

* discard changes in tmsUpload.groovy

* Remove parameters

* Restore parameters

* Change type of NodeExtDescriptorMapping to map[string]interface{}

* tmsUpload: Change type of NodeExtDescriptorMapping to map

* Resolve ioutil deprecation

* Review related changes

* Formatting

* Review related improvements

* Add tmsUtils test

* Formatting tmsUtils_test

* Remove parameters from groovy wrapper

* Remove tmsUtils_test

* Add TMS steps to fieldRelatedWhitelist

* Add integration test

* Add test to github_actions_integration_test_list.yml

* Move test helper method

* Step documentation placeholder

* Remove parameter StashContent

* Restore cmd/integrationArtifactTransport.go

---------

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2023-03-27 16:55:29 +02:00
Mayur Belur Mohan
a65df9ced6
integrationArtifactTransport Command (#4131)
* integrationArtifactTransport Command

* CodeReview Fix

* CodeReview Fix

* codereview fix

* Update documentation/docs/steps/integrationArtifactTransport.md

Co-authored-by: Srinikitha Kondreddy <srinikitha.kondreddy@sap.com>

* Update documentation/docs/steps/integrationArtifactTransport.md

Co-authored-by: Srinikitha Kondreddy <srinikitha.kondreddy@sap.com>

* CodeReview Fixes

* CodeReview FIxes

* CodeReview Fix

* Doc Fixes

* Update documentation/docs/steps/integrationArtifactTransport.md

Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>

* Doc fixes

* Doc Fixes

* CodeReview Fixes

* Doc Fixes

Co-authored-by: Linda Siebert <linda.siebert@sap.com>
Co-authored-by: Srinikitha Kondreddy <srinikitha.kondreddy@sap.com>
Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
2022-12-20 12:15:36 +01:00
Artem Bannikov
4b257377ec
[TMS] Reimplement tmsUpload step in Go (#3399)
* Initially generated tmsUpload<...> files

* First provisioning of parameters supported by tmsUpload step

* Refer to Go step from tmsUpload.groovy

* Initial client implementation

* Reverting line delimiters in tmsUpoad.groovy back to Unix ones

* Temporarily remove when-condition for Release stage

* Define useGoStep parameter in tmsUpload.groovy

* Unstash buildResult if useGoStep is true

* No unstashing and empty credentials, when using go step

* Register TmsUploadCommand in piper.go

* Cleanup groovy-related changes - they will be temporarily implemented in a different repo

* Make getting OAuth token success

* Look through the code and cleanup it a bit

* Read service key from Jenkins credentials store

* Provide initial set of unit tests for methods in /pkg/tms/tms.go file

* Minor improvements on logging response on http call error

* Check, if positive HTTP status code is as expected

* Cleanup tms.yaml file, provide additional unit test for tms.go

* Provide unit test for the case, when request body contains spaces

* Specify nodeExtDescriptorMapping parameter as of type map in tms.yaml

* Implement client method for getting nodes

* Write tests for GetNodes method

* Add GetMtaExtDescriptor client method and cover it with unit tests

* Provide first implementation for Update- and UploadMtaExtDescriptor
client methods

* Provide first implementation for Update- and UploadMtaExtDescriptor
client methods

* Provide UploadFile and UploadFileToNode client methods

* Provide tests for Update- and UploadMtaExtDescriptor client methods

* Write tests for FileUpload and FileUploadToNode client methods

* Minor corrections

* Remove some TODO comments

* Rename some of response structures

* Revert change for line delimiters in cmd/piper.go

* Add uploadType string parameter to UploadFile and UploadRequest methods
of uploader mock to reflect the changed Uploader implementation

* Start to implement execution logic in tmsUpload.go

* Changes in tms.yaml file

- remove resources from inputs in tms.yaml and implement mtaPath
parameter settings in the yaml file the same way, as it is done in
cloudFoundryDeploy.yaml
- rename tms.yaml to tmsUpload.yaml, since some generation policy
changed meanwhile

* Rename tms.yaml to tmsUpload.yaml and do go generate

* Use provided proxy on communication with UAA and TMS

* Set proxy even before getting OAuth token

* Further implementation of tmsUpload.go

* Continuation on implementing the tmsUpload.go executor

* Get mtarFilePath and git commitId from commonPipelineEnvironment, if
they are missing in configuration file + minor changes

* Implement a happy path test for tmsUpload logic

* Cover with unit tests main happy and error paths of tmsUpload.go logic

* Extend set of unit tests for tmsUpload.go

* Eliminate some TODOs, extend unit tests for tmsUpload.go

* Delete some TODOs

* Remove a couple of more TODOs from tms_test.go file

* Provide additional unit test for error due unexpected positive http
status code on upload

* Revert back line delimiters in cmd/piper.go

* Comment out file uploading calls in tmsUpload.go

* Run go generate to update generated files

* Convert line delimiters in tmsUpload.yaml to Unix ones, as well as
provide new line character in the end of the file, plus minor fix for
logging in tmsUpload.go file (pipeline complained)

* Correct description of a parameter in tmsUpload.yaml, extend unit tests
to check for trimming a slash in the end of TMS url for client methods
that do upload

* [minor] Add a comment in the test code

* Add stashContent parameter to do unstashing in tmsUpload.groovy, remove
some of the clarified TODOs

* Uncomment uploading file calls in tmsUpload.go, declare buildResult
stash in tmsUpload.yaml

* Remove clarified TODOs from the tmsUpload.go file

* Run go fmt for jenkins-library/pkg/tms

* Do not get explicitly values from common pipeline environment - all
configurations are provided in yaml file

* Remove unused struct from tmsUpload_test.go

* Run go fmt jenkins-library\pkg\tms

* Revise descriptions of parameters provided in tmsUpload.yaml file

* Specify STAGES scope for tmsUpload parameters

* Provide STAGES scope for the tmsUpload parameters, provide default value
for stashContent parameter

* Remove trailing space from tmsUpload.yaml

* Provide unit tests for proxy-related changes in http.go file

* Improve proxy implementation in tmsUpload.go file

* Make tmsServiceKey again a mandatory parameter

* Run go generate command to make the generated files correspond the yaml
state

* Change line delimiters back to Unix ones (were switched while resolving
the conflicts)

* Remove trailing spaces from tmsUpload.yaml

* Minor change in a comment to trigger pipelines with commit

* Improve checks for zero-structs and for empty maps, as well as use
different package to read files in the tests

* Revert line endings in http.go

* Revert comments formatting changes in files that do not belong to the tmsUpload step
2022-08-30 10:16:09 +02:00
Mayur Belur Mohan
5931415d9c
ApiProviderList Command (#3879)
* ApiProviderList Command

* Metadata Fix

* Metadata Fix

* CodeReview Fixes

* Documentation Fixes

* unit test fix

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2022-07-07 15:48:59 +03:00
sumeet patil
78cf40799e
feat(codeql): new codeql piper step (#3765)
* codeql piper step
2022-06-24 09:04:24 +02:00
Oliver Feldmann
810d197665
[ANS] Step implementation (#3764)
* Add ans implementation

* Remove todo comment

* Rename test function

Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>

* Better wording

Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>

* Add reading of response body function

* Use http pkg ReadResponseBody

* Check read error

* Better test case description

* Fix formatting

* Create own package for read response body

* Omit empty nested resource struct

* Separate Resource struct from Event struct

* Merge and unmarshall instead of only unmarshalling

* Improve status code error message

* Remove unchangeable event fields

* Separate event parts

* Change log level setter function

* Restructure ans send test

* Revert exporting readResponseBody function

Instead the code is duplicated in the xsuaa and ans package

* Add check correct ans setup request

* Add set options function for mocking

* Review fixes

* Correct function name

* Use strict unmarshalling

* Validate event

* Move functions

* Add documentation comments

* improve test

* Validate event

* Add logrus hook for ans

* Set defaults on new hook creation

* Fix log level on error

* Don't alter entry log level

* Set severity fatal on 'fatal error' log message

* Ensure that log entries don't affect each other

* Remove unnecessary correlationID

* Use file path instead of event template string

* Improve warning messages

* Add empty log message check

* Allow configuration from file and string

* Add sourceEventId to tags

* Change resourceType to Pipeline

* Use structured config approach

* Use new log level set function

* Check correct setup and return error

* Mock http requests

* Only send log level warning or higher

* Use new function name

* One-liner ifs

* Improve test name

* Fix tests

* Prevent double firing

* Reduce Fire test size

* Add error message to test

* Reduce newANSHook test size

* Further check error

* Rename to defaultEvent in hook struct

* Reduce ifs further

* Fix set error category test

The ansHook Fire test cannot run in parallel, as it would affect the
other tests that use the error category.

* Change function name to SetServiceKey

* Validate event

* Rename to eventTemplate in hook struct

* Move copy to event.go

* Fix function mix

* Remove unnecessary cleanup

* Remove parallel test

The translation fails now and again when parallel is on.

* Remove prefix test

* Remove unused copyEvent function

* Fix ifs

* Add docu comment

* Register ans hook from pkg

* register hook and setup event template seperately

* Exclusively read eventTemplate from environment

* setupEventTemplate tests

* adjust hook levels test

* sync tests- wlill still fail

* migrate TestANSHook_registerANSHook test

* fixes

* Add ans send event step

* Fix tests

* Add groovy wrapper

* Add groovy wrapper test

* Fix function names

* Reduce ifs

* Fix docu

* We always set the timestamp

* Validate event

* Test unknown field in json

* Make test list test

* Set all event fields as separate parameters

* Generate and fix code

* Review fixes

* Format test file

* Format go code

* Fix common steps tests

* Print event to console if verbose

Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
Co-authored-by: Roland Stengel <r.stengel@sap.com>
2022-06-21 15:45:07 +02:00
Mayur Belur Mohan
cdea4b7713
ApiProxyList Command (#3794)
* ApiProxyList Command

* CodeReview Fixes

* CodeReview Fixes

* CodeReview FIxes

* CodeReview Fixes

* CodeReview FIxes

* CodeReview Fixes

* fixing unit test

* doc fixes

* Update documentation/docs/steps/apiProxyList.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* CodeReview Fixes

* CodeReview Fixes

* CodeReview Fixes

* codereview fix

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2022-06-21 13:08:00 +02:00
Julian Schmitt
dd0aae6ded
Create azureBlobUpload (#3766)
* add Step azureBlobUpload

* add azure sdk and unit tests

* add Documentation

* fix Groovy Wrapper

* adopt the requested changes from awsS3Upload

* fix lint tests

* downgrade azure sdk to go 1.17

* multiple fixes e.g. use of temporary files for tests

* fix tests

* Update cmd/azureBlobUpload.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update cmd/azureBlobUpload.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update documentation/docs/steps/azureBlobUpload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update documentation/docs/steps/azureBlobUpload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update documentation/docs/steps/azureBlobUpload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update documentation/docs/steps/azureBlobUpload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* requested changes

* use latest version of azure sdk after update to go 1.18

* change staticcheck from 1.1.0 to 1.2.0

* try to fix lint test by pre-compiling go 1.18

* fix caching for lint test

* improve error handling by dividing runner

* improve error handling and add validation

* multiple naming fixes

* add new test for unmarshalling JSON-Structs

* Update cmd/azureBlobUpload_test.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update cmd/azureBlobUpload_test.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update cmd/azureBlobUpload_test.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* fix JSON unmarshall test

* Update documentation/docs/steps/azureBlobUpload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update cmd/azureBlobUpload_test.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Update cmd/azureBlobUpload.go

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* fix uploadFunc

Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2022-06-15 09:41:02 +02:00
Daniel Mieg
cda6e71ab9
Add ABAP step: createTag (#3633)
* Cloud Platform -> BTP

* Initial generation of new step

* add flag

* wip

* Fix warnings

* Add command

* Added Tag Decription

* Add status check

* Improve handling

* Improve handling

* Add test for happy path

* Add test reports to gitignore

* Add second test

* Improve createTag

* Add testcase

* Adaptions

* Add test

* Update cmd/abapEnvironmentCreateTag.go

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* Update cmd/abapEnvironmentCreateTag.go

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* Adapt error handling

* Improve coding

* Add info

* Disallow repositories and repositoryName at the
same time

* Regenerate

* Adapt to feedback

* Update cmd/abapEnvironmentCreateTag.go

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* Update cmd/abapEnvironmentCreateTag.go

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>
2022-05-23 15:15:22 +02:00
Julian Schmitt
6714794066
Create awsS3Upload (#3737)
* add Step awsS3Upload

* fix JSON Read

* fix groovy wrapper

* change credentials to secret text

* Change credentials type to token

* add cleanup for environment variables

* Add AwsS3UploadCommand to piper.go

* add documentation of awsS3Upload

* Fix JSON String Example in Documentation

* add the Upload of whole directories

* add Logging to awsS3Upload.go

* imporve Logging

* fix and improve unit tests

* fix non-existing-filepaths-Bug

* fix windows filepaths

* remove ... from logging

* change step description

* fix PR Tests

* remove redundant code

* try to run tests sequentially

* fix file.Close

* executed go mod tidy

* requested changes

* fix comments and awsCredentials

* Update documentation/docs/steps/awsS3Upload.md

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* use temporary files for unit tests

* executed go generate

Co-authored-by: ffeldmann <f.feldmann@sap.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
2022-05-18 10:37:54 +02:00
TheShifter
188e743f7b
PythonBuild: Implementation of pythonBuild step (#3483)
* Implementation of pythonBuild step

* minor update and refactoring

* minor update

* add integration test and test project to testdata dir

* remove generated build data dir

* Rewrite some logic. Minor fix in integration tests for python

* Add new input parameters to pythonBuild.yaml

* rewrite logic
remove some checks

* rollback

* resolve merge conflict in piper.go
Update logic in python build. Create bom now works fine

* remove duplicate line

* refactoring fix

* resolve comment. Remove install build and change build command. Change twine upload command

* add groovy wrapper for pythonBuild step

* Rewrite tests. Remove some cheks from pythonBuild.go

* add some test to pythonBuild_test.go

* Add some parameters and credentials to the pythonBuild.groovy

* fix issue in unit tests

* add pythonBuild to fieldRelatedWhitelist

* update integration test for pythonBuild

* add imports

* update integration tests and add a new one

* minor fix

* fix some issues in integration tests

* update integration tests. Make it works again

Co-authored-by: Anil Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2022-03-29 19:01:44 +02:00
ffeldmann
ccc1c976ee
fix(orchestrator) usage of correct env variables (#3650)
* Reorders getApiInformation, changes variables to get start time, adjusts and adds test cases
* Changes the way to get apiInformation and reduces number of requests
* Changes getting pipeline start time from correct env variable
* Refactors getApiInformation functionality
* Adds GetBuildReason() for Azure and Jenkins
* Updates JobURL for ADO
2022-03-28 09:52:15 +02:00
maheshsrikrishnan
64a00c540a
New piper step APIProviderDownload contribution (#3349)
* added store file function in cpi common utils

Change-Id: Ia429a2792266e082d139025a71799d21c30a7df9

* Added api provider steps

Change-Id: Icd2829a91db4c53d0de2330822d2b33933973868

* Update apiProviderDownload.yaml

* fixed yaml JLINT issue

Change-Id: Iac974abc30fa00e68c0177072b93716b0af5e0c5

* Removed trailing spaces

Change-Id: I927e9314fce6e9cab68d6b97577c7c96bb2bddad

* Resolved common steps groovy script conflict

Change-Id: I3ad144b618e1c77953aaeccaa5bf7309aff77ca9

* Change for conflict resolution

Change-Id: Ic955833eca844f090b7983f99f9d3649ebb981c7

* Fixed method name and its corresponding implementation

Change-Id: I465c1f1d5306bb978386de9efca3c521e385b89c

* Moved re-usable function to commonUtils package

Change-Id: Ide06462b01caeb2bf438ad7661e01c15bf8e8e24

* Changed the implementation to use existing writeFile method

* Fixed review comments on documentation and test structuring

Change-Id: Ifebd2f4b50754b2097b2d564fb3cc37c433ef6c9

* Fixed documentation alignment issues

* Fixed spaces issue

Change-Id: I834bd94e01bce72e7f81ab49ba32671c91c66ca9

* Documentation removed extra spaces

Change-Id: I9a639d76ed9b81c870f18349504044bb70753b52

* Fixed doc build issue

Change-Id: I96c3e15e73834b64f8b8e3432ce59f6b037f93fd

* Fixed documentation build issues

Change-Id: I7fca2ba69bc7b7298ee300ccd1ae16a6238dc96b

* Re-generated code for fixing build failure

Change-Id: I22b7ee6162f643d9f3b60f6a33eb7858927182a0

* Adopted file utils & mock

Change-Id: Ic46462003527f41df64395a5a615c19bf374e8ef

* Removed ioutil call in the test & adopted error variable names

* Removed commented lines

Change-Id: I99a12e39bc04323e9c19f1409d97eeca267e6fdb

* Added test for asserting file download and adopted error variables

Change-Id: I49463a3b75987bf68f5261d45602d2d7bd960a05

* Added download path assertion positive & negative case

Change-Id: Ieee461c3973b9dfa8f395dc936e4241ff9694c7b

* Modified tests with DownloadPath variable

Change-Id: Iaf14c9ea1a8242b6c8d8e9e4fac8c23d9c1b3a74

* Added testcase to validate file content

Change-Id: I21aed481b433450c3b536dbb29d45291f61848d8

* Refactored test for file content check to avoid failures

Change-Id: I3b4fe9a0de678f437fd4cc0a8203ae9434d9fa8e

* Removed auto-generated comments

Change-Id: I86c4ac3e7e4476a75d6cbed58826ec1f3278d7d2

* Fixed documentation review comments

Change-Id: I4faf31473b53fc53a5517d418c343bf7320eec55

* Fixed documentation indentation

Change-Id: I386f046cf4e10ee6deb5a81fcfc8c430c97086c8

* Fix build issue

Change-Id: I61a829cabaf03ffd5e77cddc594486a650118fa3
2022-03-09 13:07:23 +01:00
Mayur Belur Mohan
9a3b800b9d
ApiKeyValueMapUpload Command (#3407)
* ApiKeyValueMapUpload Command

* include command in metadata

* TestCase Fixes

* CodeReview Fixes

* CodeReview Fixes

* Code Review Fixes

* CodeReview Fixes

* CodeReview Fixes

* CodeReview FIxes

* CodeReview Fixes

* Documenation change

* documentation fix

* Documentation Fix

* Documentation Fix

* documentation Fix

* CodeReview Fixes

* CodeReview Fixes

* Revert changes

* Documentation Fix

* CodeReview FIxes

* Doc Fixes

* Code Review Fixes

* Code Review Fixes

* CodeReview FIxes

* Documentation Fix

* Documentation Changes

* Documentation Fix

* codereview fix

* Documentation Fix

* CodeReview Fixes

* CodeReview Fix

* Documentation FIx

* doc fix

* Doc Fix

* Documentation Fix

* codereview fix

* revert fix

* Code Review Fix

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2022-03-07 11:03:44 +01:00
Rinita Asani
cd2ae89229
Local scopeg cts execute abap unit test (#3563)
* Adjust local changed objects with commit history

* fix the righ history entry

* fix for local packages commit history

* Adapt unit tests and fix for retry after 500

* Rename gctsExecuteABAPUnitTests to gctsExecuteABAPQualityChecks

* Add unit test file for gCTSExecuteABAPUnitTest

* add a test step

* gcts Test change files

* Add unit test for gCTS test

* adapt gctsTEST

* deletegcts

* command

* maintain old step

* Adapt changes to documentation

* fix for go generate
2022-02-22 09:58:22 +01:00
Vitalii Sidorov
93e3801945
feat: add step for Helm execution (#3419)
* Implement helm step

* Create kubernetes package

* Refactoring helm.go

* Add package, test commands

* Add test for helm package

* Add tests for helm.go

* Add tests for helm.go

* Add tests for utils.go

* Add tests for helmExecute.go

* small fix

* Add helm lint

* small fix

* small fix

* Fix according to comments

* Fix test

* small fix

* Add helm add function

* Changes according to new comments

* Add helm push

* Add unit tests

* Add tests for helmExecute

* Add small fix

* small fix

* small fix

* Move DeployUtilsBundle from kubernetesDeploy to kubernetes package

* small fix

* small fix

* Add unit-tests

* Fix

* Update resources/metadata/helmExecute.yaml

* Update resources/metadata/helmExecute.yaml

* Add helm chart server parameterization

* small fix

* small fix

Co-authored-by: “Vitalii <“vitalii.sidorov@sap.com”>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2022-02-10 10:25:03 +01:00
Daniel Bernd
0d8bd7dc52
Add step abapEnvironmentPushATCSystemConfig (#3426)
* Checkin Go File - Push ATC Config

* ATC Push Config - Testfile

* Initial generated go files for ATC System Configuration

* ApiProxyUpload Command (#3295)

* ApiProxyUpload Command

* Code Review Fixes

* CodeReview Changes

* CodeReview Fixes

* YAML fixes

* CodeReview Fix

* Code Review Fixes

* CodeReview Fixes

* Code Climate Fixes

* Code Review Fixes

* Code Review Fixes

Co-authored-by: Roland Stengel <r.stengel@sap.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>

* Fetch Xcrsf-Token

* Update  abapEnvironmentPushATCSystemConfig.yaml

* docs: fix urls (#3400)

* fix stash and adapt unit tests accordingly (#3403)

* Push ATC System Configuration - File read, send request, parse response

* fetchATCXcsrfToken

* fix(whitesourceExecuteScan): added missing credential declaration for the docker config (#3404)

* insert command in piper.go

* feat(cnbBuild): remove docker config after parsing (#3417)

Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>

* more funcs

* more funcs

* Revert "feat: Add debug output to getConfig (#3341)" (#3396)

This reverts commit b760bf48e7.

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* feat (kubernetesDeploy) allow secret creation in cluster with kubectl as deploy tool (#3398)

* defaulting deocker config json location

* function change

* harmonising secret creation via json config apply

* adding the env path to kaniko

* env var

* adding path

* adding path

* adding path

* rolling back changes for path

* adapting condition for secret creation

* conditions based on username and password only

* fix unit test as per new secret creation

* update documentation with regards to secret creation

* fixing yaml lint empty line

* fixing trailing line from kaniko yaml

* error condition when path of the docker config json file is not found

Co-authored-by: anilkeshav27 <you@example.com>

* readPipelineEnv -> Serialize Directly to os.Stdout (#3180)

* Serialize CPE Directly to os.Stdout

* Response Error Handling 1

* Add easy mode for AUnit & ATC (#3389)

* remove mandatory flag from config files

* Enable repo.yml as config

* Adapt to merge

* Refactoring

* Refactoring

* avoid panic

* Add comments

* Add easy mode for atc

* Add tests

* Add test

* Refactor

* Add test for MPS

* Updates

* Rename functions

* Add files to gitignore

* Rename

* Renaming

* Renaming

* Renaming

* Improve error messages

* Update documentation

* Add logging

* Rename

* Extend gitignore

* fix(sonar): fix file patterns for gcs upload (#3406)

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* testing piper forks  (#3420)

* developer doku update on how to test with forked repos
Co-authored-by: rosemarieB <45030247+rosemarieB@users.noreply.github.com>

* Error Messages parsing

* Update yaml parameter

* feat (npmExecuteScripts) create seperate npmrc file for publish to private repo (#3422)

* creating new npm rc file

* publishing to registry staging

* exposing base64 version of env variables

* changing encoding param

* fixing unit test for the new path

* debugging env var

* remove debug message

* update docu

* changing new npmrc file name

* adding new npmrc to ignore

* adding new npmrc to ignore

Co-authored-by: anilkeshav27 <you@example.com>

* minor

* small changes

* small changes

* UNIT Test

* Unit Test 2

* Update .gitignore

Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>

* Update abapEnvironmentAddons.md: Unable to update target vectors with unchanged SWC version (#3392)

* Update abapEnvironmentAddons.md

* Update documentation/docs/scenarios/abapEnvironmentAddons.md

Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>

* Update abapEnvironmentAddons.md

Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>

* Returning earlier & parseOdataResp 1

* new Parameter for overwrite existing

* feat(cnbBuild): preserve maven test results in the workspace (#3429)

Co-authored-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>

* Update Introduction (#3433)

* Check Existence of configuration

* Quote strings in addon.yml (#3435)

* including vaultSecretFile to parameter generator docu (#3436)

Co-authored-by: anilkeshav27 <you@example.com>

* fix: wrong command line format for multiple extension files (#3434)

* feat(cnbBuild) containerImageName will be defaulted if possible (#3437)

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
Co-authored-by: Johannes Dillmann <j.dillmann@sap.com>
Co-authored-by: Sumit Kulhadia <sumit.kulhadia@sap.com>

* Corr. check existence

* check existence 2

* existence handling 2

* refinement existence handling & param rename

* config yaml change - no mandatory for bool param

* PATCH for existing Configuration

* fix: correct typo in log message (#3439)

Closes #3438

* feat(protecode): add versioning model (#3373)

* changes to detectExec before master merge

* changes for detectExecuteScan

* self generated code added

* fix syntax errors and update docu

* added unit tests for fail and Group

* fix failOn bug

* add Groups as string array

* add Groups as string array

* tests and validation for groups, failOn

* Updated docs and added more tests

* documentation md files should not be changed

* Handle merge conflicts from PR 1845

* fix merge errors

* remove duplicate groups, merge error

* adding buildCode and buildTool as params

* switching build options

* building maven modules

* parameter correction

* parameter correction

* gnerate with new build parameter

* adding comments

* removing piper lib master and modifying goUtils to download 1.5.7 release

* first cleaning then installing

* multi module maven built

* multi module maven built removing unwanted code

* multi module maven built moving inside switch

* testing

* modifying the default use case to also call maven build

* modifying the default use case to also call maven build wih --

* corrected maven build command

* corrected maven build command with %v

* skipping test runs

* testing for MTA project with single pom

* adding absolute path to m2 path

* clean up

* adding switch for mta and maven and removing env from containers

* commiting changes for new detect step

* correting log message

* code clean up

* unit tests changes to detectExecute

* basic tests for new change

* restoring piperGoUtils to download correct piper binary

* code clean up

* code clean up

* protecodeExecuteScan :: versioning model draft - 1

* protecodeExecuteScan :: version model draft-2

* protecodeExecuteScan :: changing filename and version concatenation

* protecodeExecuteScan :: update documentation

* protecodeExecuteScan :: double URL encoding has been corrected & console messaging improved

* protecodeExecuteScan :: fixed Go/generate validation fail

* protecodeExecuteScan :: fixing failed unit tests

* protecodeExecuteScan :: Version field added

* protecodeExecuteScan :: Version field add => minor changes

* protecodeExecuteScan :: Version field add => fixing tests

Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>

* GCS Upload fixes (#3387)

* Fixed argument type in persist function

* Fixed gcp upload to be usable in internal piper

* Fixed import of packages

* Updated logs

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Unit Test 2

* Unit Tests

* Add gcs upload to mavenExecuteIntegration step (#3432)

* Add gcs upload to mavenExecuteIntegration step

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Check times refine

* Permwrite 0644

* disable http retry (#3447)

* Update maxWaitTime timeout for WhiteSource (#3284)

* Update scanPolling.go

Changing maxWaitTime from 15 to 30 to overcome WhiteSource results reflection in the backend issue.

* Update configHelper.go

* Reset configHelper changes to fix PR 3284

 Committer: raghunathd8

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: raghunathd8 <root@docker-evaluation.openstack.eu-nl-1.cloud.sap>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>

* removed trailing spaces

* yaml Update

* minor mock for token & Unit Tests

* rename Parameter Patch

* lil refactor & Unit Test

* fixes PATCH & more Unit Tests

* add log at certain points

* not writing configuration file back

* feat(malwareExecuteScan): refactoring and docker support (#3421)

* feat(malwareExecuteScan): add support for scanning docker images

* refactoring

* print out finding if available

* generate toolrecord for malware scan

* persist scan report

* docs

* fix

* fix

* rollback cmd/init_unix.go

* auhenticated pull

* fix

* fix: report shall be consistent with the api model

* gcs upload

* fix linter

* Fix potential nil reference (#3460)

* Add gcs upload to karmaExecuteTests step (#3431)

* Add gcs upload to karmaExecuteTests step

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Add gcs upload to gaugeExecuteTests step (#3410)

* Add gcs upload to gaugeExecuteTests step

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Add gcs upload to newmanExecute step (#3408)

* Add gcs upload to newmanExecute step

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* feat(GitHub): fetch commit statistics (#3381)

* fetch GH statistics

* move GH and Sonar integration tests to own files

* fix imports

* add integration test case

* add result type

* Apply suggestions from code review

* feat: add command to fetch default files (#3445)

* some ideas..

* Add getDefaults command (WIP) (#3444)

* add getYAML function for configs

* create getDefaults command(based on getConfig)

* add getDefaults command to CLI

* read defaults files, using github tokens as well

* write defaults to stdout as JSON object with YAMLs embedded

* catch case where no input files are given

* actually write output to file if outputFile is specified

* mark defaultsFile flag as required

* add basic tests

* add output (string)  test

* adapt generateDefaults() to return output (used for test of previous commit)

* Changes to getDefaults() JSON output (#3449)

* change JSON output to contain separate fields

* filename -> filepath

* Apply suggestions from code review

* Apply suggestions from code review

* Update pkg/config/config_test.go

Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>

* BATCH request preparation

* Add gcs upload to mtaBuild step (#3405)

* Add gcs upload to mtaBuild

* Patterns was updated

* go generate

* removed patterns related to sapGenerateEnvironmentInfo step

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Add gcs upload to mavenBuild step (#3394)

* Add gcs upload to mavenBuild step

* Patterns were updated

* go generate

* removed patterns related to sapGenerateEnvironmentInfo step

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* feat(gradleExecuteBuild): add stage scope to path parameter, fix project dir exist issue (#3401)

* add stage scope to path parameter, fix project dir exist issue

* fix unit test for gradleExecuteBuild

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Add gcs upload to uiVeri5ExecuteTests step (#3409)

* Add gcs upload to uiVeri5ExecuteTests step

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Address further nil references (#3462)

* Address further nil references

* Message text

* Final checks

* fix (npmExecuteScripts) allow npm pack before npm publish (#3455)

* adding config to piperNpmr

* scope in cli

* adding scope to repo url and npmrc

* publish to scoped

* removing scope

* changing scope position

* adding scope to userconfig

* adding registry=

* pack and then tar

* not removing tmp folder

* adding flag

* pack before publish

* adding log

* debug

* debug with change directory

* publishing created tar ball

* debug

* üath

* adding main npmrc

* renaming old npmrc file

* error renaming old npmrc file

* renaming err

* correcting npmrc file path

* renaming file back to original

* current working directory

* renaming the npmrc file

* avoiding change directory

* with current working dir

* adding dot

* renaming npmrc and defer removal

* rename files

* Update pkg/npm/publish.go

* Update pkg/npm/publish.go

Co-authored-by: anilkeshav27 <you@example.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Add testing for helm during acceptance stage (#3402)

* Add kubernetesDeploy to Acceptance

* Add more kubernetesDeploy

* Add helm tests

* Change documentation

* Fix docu

* Change generated

* Add tests

* Add groovy tests

* Fix tests

* Change tests

Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>

* Batch Request for Patches

* Fix(gctsDeploy) : add client in config url, disable retry for create and pull (#3464)

* minor Unit Test correction

* lil cleanup

* lil refactoring

* removed nuneeded linebreaks

* refactoring Command on Entity in Batch

* some Unit Tests for Build Batch Request

* more Unittests - Build Batch Request

* fix(fortifyExecuteScan): User assignment based on PR ownership (#3472)

* Debug PR user details

* Check association

* Change to login

* Fix PR creator assignment

* Improve docs

* Fix test

* Update gradle version (#3476)

* Update gradle version

* Update generated file

* small correction Text & Test

* chore(kubernetesDeploy): add telemetry for deployTool (#3469)

* chore(kubernetesDeploy): add telemetry for deployTool

* Update cmd/kubernetesDeploy.go

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* rename Parameter patchIfExisting

* fix (mtaBuild) keep mtar artifact name in synch with maven gav (#3454)

* keep mtar name in synch with maven gav

* retaining slashes

* keeping dots in artifact name

Co-authored-by: anilkeshav27 <you@example.com>

* Update image for gradle build (#3479)

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* Last Changed not Zero check added

* Added pseudo_comment_policy in struct

* Unit Test adaption

* smaller refactoring, Unit Test, removed doubles

* fix(cnbBuild) do not add complete path to telemetry (#3487)

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>

* Add limitation that table keys are exported generically (#3490)

* Add limitation that table keys are exported generically

* Fix formatting

* Clarify delta calculation of different package types (#3482)

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* Add troubleshooting for missing add-on registration (#3491)

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* feat (Documentation) documentation for mavenBuild and npmExecuteScripts that consume dependencies from a private repo. (#3484)

* build depdency docu for maven and npm

* removing trailing white space in yaml

* Update resources/metadata/mavenBuild.yaml

* relative url for vault and mta docu

* running go generate

* keeping vault relative path

* go generate

* reverting to global paths

* go generate

* wild card for a higher level dir

* searching on top folder only

* relative level above

* pointing to infrastructure

* correcting links

Co-authored-by: anilkeshav27 <you@example.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* fix(kanikoExecute): tmp dir (#3478)

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* feat(golangBuild): add support for publishing binaries (#3495)

* fix(golangBuild): keep track of the artifactversion in binary names (#3498)

* fix(golangBuild): keep track of the artifactversion in binary names

* improve the naming scheme

* fix (build dependency) typo corrections (#3494)

* typo corrections

* trailing white space

* go generate

Co-authored-by: anilkeshav27 <you@example.com>

* Revert "fix(golangBuild): keep track of the artifactversion in binary names (#3498)" (#3499)

This reverts commit ec783b0da6.

* sonarqube coverage: additional metrics (#3465)

* sonarqube coverage: additional metrics

* sonarExecuteScan: add lines of code and language distribution to sonarscan.json

* sonarExecuteScan: consider branch in componentService requests

* SonarQube: Do not omit empty values in SonarCoverage

* sonarExecuteScan: Add integration tests for ComponentService getLinesOfCode

* fix tests

* sonarExecuteScan: use pullRequest in componentService

Co-authored-by: I550025 <r.kloe@sap.com>
Co-authored-by: Marc Bormeth <marc.bormeth@sap.com>

* fix(golangBuild): targetRepositoryUrl (#3502)

* fix(golangBuild): targetRepositoryUrl

* regen

* fix(httpclient): folder permissions (#3506)

* fix(golangBuild): use PUT instead of POST (#3504)

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* feat(golangBuild): publish to subfolder (#3503)

* feat(golangBuild): publish to subfolder

* edge case handling

* fix

* fix(golangBuild): expect status 201 instead of 200 (#3508)

* feat(kanikoExecute): allow building multiple images (#3443)

* feat(kanikoExecute): allow building multiple images

* enhance tests

* chore: allow running tests in parallel

* small fixes

* fix: fix destination bug

* update formatting and defaults

* fix yml formatting

* chore: change cpe parameter names

* chore: improve variable naming

* Changes GetJobName value to repositoryname (#3509)

* Update npmExecuteScripts step (#3211)

* Update npmExecuteScripts step

* Fixed failing build

* Fixed path issue

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Christian Volk <christian.volk@sap.com>

* Ignore violations if failOn is NONE (#3513)

* fix(sonarExecuteScan): error handling for report metrics (#3512)

* Generic step   input from addon descriptor (#3467)

* enhance build step to run based on addon.yml in steampunk environment

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>

* feat(golangBuild): add BOM creation (#3496)

* Added BOM creation

* Fix cyclonedx install

* undo config.yml changes

* metadata was updated

* Update golangBuild.go

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* If statement dearrangement

Co-authored-by: Daniel Bernd <93763187+danManSAP@users.noreply.github.com>
Co-authored-by: Mayur Belur Mohan <68264167+mayurmohan@users.noreply.github.com>
Co-authored-by: Roland Stengel <r.stengel@sap.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>
Co-authored-by: Jesse Awan <jesse.awan@sap.com>
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
Co-authored-by: Marc Bormeth <marc.bormeth@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Anil Keshav <anil.keshav@sap.com>
Co-authored-by: anilkeshav27 <you@example.com>
Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
Co-authored-by: Siarhei Pazdniakou <siarhei.pazdniakou@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Peter Persiel <6087940+peterpersiel@users.noreply.github.com>
Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
Co-authored-by: Marcus Holl <marcus.holl@sap.com>
Co-authored-by: Johannes Dillmann <j.dillmann@sap.com>
Co-authored-by: Sumit Kulhadia <sumit.kulhadia@sap.com>
Co-authored-by: Nikolay Grechanov <nikolay.grechanov@sap.com>
Co-authored-by: Umidjon Urunov <79094563+UmidjonUrunov@users.noreply.github.com>
Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Sarat Krishnan <78093145+sarat-krk@users.noreply.github.com>
Co-authored-by: Raghunath Deshpande <raghunath.deshpande@sap.com>
Co-authored-by: raghunathd8 <root@docker-evaluation.openstack.eu-nl-1.cloud.sap>
Co-authored-by: Christian Volk <christian.volk@sap.com>
Co-authored-by: Eugene Kortelyov <eugene.kortelyov@sap.com>
Co-authored-by: Linda Siebert <39100394+LindaSieb@users.noreply.github.com>
Co-authored-by: Thorsten Duda <thorsten.duda@sap.com>
Co-authored-by: Matthias Scudlik <matthias.scudlik@gmail.com>
Co-authored-by: I550025 <r.kloe@sap.com>
Co-authored-by: ffeldmann <f.feldmann@sap.com>
Co-authored-by: ManjunathMS35 <manjunathms35@gmail.com>
Co-authored-by: Matthias Scudlik <matthias.scudlik@sap.com>
Co-authored-by: rosemarieB <45030247+rosemarieB@users.noreply.github.com>
2022-02-08 16:14:29 +01:00
Christopher Fenner
ebcbb73b46
feat: add command to fetch default files (#3445)
* some ideas..

* Add getDefaults command (WIP) (#3444)

* add getYAML function for configs

* create getDefaults command(based on getConfig)

* add getDefaults command to CLI

* read defaults files, using github tokens as well

* write defaults to stdout as JSON object with YAMLs embedded

* catch case where no input files are given

* actually write output to file if outputFile is specified

* mark defaultsFile flag as required

* add basic tests

* add output (string)  test

* adapt generateDefaults() to return output (used for test of previous commit)

* Changes to getDefaults() JSON output (#3449)

* change JSON output to contain separate fields

* filename -> filepath

* Apply suggestions from code review

* Apply suggestions from code review

* Update pkg/config/config_test.go

Co-authored-by: Jordi van Liempt <35920075+jliempt@users.noreply.github.com>
2022-01-24 13:55:28 +01:00
Mayur Belur Mohan
f27cb4e482
ApiProxyUpload Command (#3295)
* ApiProxyUpload Command

* Code Review Fixes

* CodeReview Changes

* CodeReview Fixes

* YAML fixes

* CodeReview Fix

* Code Review Fixes

* CodeReview Fixes

* Code Climate Fixes

* Code Review Fixes

* Code Review Fixes

Co-authored-by: Roland Stengel <r.stengel@sap.com>
Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2021-12-28 09:31:50 +01:00
Eugene Kortelyov
4d787ded6d
gradleExecuteBuild initial commit (#3337)
* gradleExecuteBuild initial commit

* fmt fixes for metadata_generated

* flow updates, some tests were added

* add gradle path alias, add stage scope to task parameter

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-12-22 15:34:36 +01:00
Christopher Fenner
51feacadbf
chore: align Vault naming (#3372)
* Align Vault naming

* remove duplicate debug  output

* correct message

* align vault naming

* Update vault.md
2021-12-21 11:52:10 +01:00
Siarhei Pazdniakou
cd243ee542
feat(gcs): allow upload to gcs from steps (#3034)
* Upload reports to Google Cloud Storage bucket

* Added tests. Made fixes

* Update step generation. GCS client was moved to GeneralConfig

* Code was refactored

* Fixed issues

* Fixed issues

* Code correction due to PR comments

* Improved gcs client and integration tests

* Integrated gcp config. Updated step metadata

* Fixed issues. Added tests

* Added cpe, vault, aliases resolving for reporting parameters

* Added tests

* Uncommented DeferExitHandler. Removed useless comments

* fixed cloning of config

* Added comments for exported functions. Removed unused mock

* minor fix

* Implemented setting of report name via paramRef

* some refactoring. Writing tests

* Update pkg/config/reporting.go

* Update cmd/sonarExecuteScan_generated.go

* Apply suggestions from code review

* Update pkg/config/reporting.go

* Update pkg/config/reporting.go

* fixed removing valut secret files

* Update pkg/config/reporting.go

* restore order

* restore order

* Apply suggestions from code review

* go generate

* fixed tests

* Update resources/metadata/sonarExecuteScan.yaml

* Update resources.go

* Fixed tests. Code was regenerated

* changed somewhere gcp to gcs. Fixed one test

* move gcsSubFolder to input parameters

* fixed removing valut secret files

* minor fix in integration tests

* fix integration tests

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2021-12-15 15:07:47 +01:00
Oliver Nocon
9a78fabc89
feat(golangBuild): add new step for building go (#3178)
* feat(golangBuild): add new step for building go

* chore(golangBuild): increase test coverage

* remove indirect dependencies

* cleanup go.sum

* chore: remove trailing spaces

* chore(golangBuild): cleanup params, add groovy wrapper

* fix: update docker options

* update docs

* update installation according to https://golang.org/doc/go-get-install-deprecation

* fix: update installation

* update groovy test exclusion

* Update vars/golangBuild.groovy

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* update branch

* address PR feedback

* fix compilation error

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2021-12-06 16:17:59 +01:00
rosemarieB
e90856d5bf
Generic build step (#3323)
* new step abapEnvironmentBuild

* Update piper.go

* Update abapEnvironmentBuild.go

* update yaml file

* Logging for debugging

* Update abaputils.go

* Update connector.go

* assigning connector

* delete debugging logging

* Update abapEnvironmentBuild.go

* certificate to yaml

* Update abapEnvironmentBuild.go

* add scope

* Update abapEnvironmentBuild.go

* Update abapEnvironmentBuild.yaml

* change certificate name in yaml

* test my new gitscript

* logging for debugging

* debugging...

* adding options to client.

* skip verification

* debugging

* debugging...

* switch of transportskipverification

* changing connector return

* deleting additional set options

* fixed timeout error

* adding certificate

* testing without certificate set

* testing with certificate set

* download, publish and value logic

* write values to cpe

* logging

* adding condition on string length

* change publishmethod and some logging

* change download method -> using references

* evaluation of parameter for download

* add case for empty string

* adding unittests

* Update mockClient.go

* make abapEnvironmentBuildUtilsBundle powerful

* refactor abapEnvironmentBuild into pieces

* check error message

* check error message 2

* check error message 3

* check error message 4

* remove check error message

* cleanup

* adding unittests

* unittests and docu

* docu

* docu

* Update abapEnvironmentBuild.md

* removing trailing spaces and adding empty lines in docu

* Update abapEnvironmentBuild.md

* fixing unittest and PR recommen

* Update abapEnvironmentPipelineStageBuild.groovy

* Update abapEnvironmentPipelineStageBuild.groovy

* Update abapEnvironmentPipelineStageBuild.groovy

* Update abapEnvironmentPipelineStageBuild.groovy

* changes derived from pull request

Co-authored-by: tiloKo <70266685+tiloKo@users.noreply.github.com>
2021-12-06 14:43:37 +01:00
Eugene Kortelyov
d395b362ed
feat: step to execute shell scripts (#3196)
* shell executor initial commit

* functionality updates

* changes in logging implementation (using internal logging), changes in execution

* remove unused field

* remove duplicate from code

* update vault flow and remove unnecessary params

* update generated step file

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-19 10:05:39 +01:00
ffeldmann
42b92d1bfe
Changes for Pipeline Reporting (#3213)
* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetRequest function which holds the response in memory (not saved to disk)

* Implements GetLog() function for ADO, adds function to read PipelineRuntime

* PAT has been revoked

* Changes http package, s.t. if password only is required basic auth works too

* Adds env variable for azure token, error handling in case of unauthenticated/nil response

* Adds logging output in case env variable can not be read and fallback variable needs to be used

* Adds usage of environment variables for auth, uses jenkins api

* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function

* Adds initaliziation function for orchestrator authetnication

* Adds settings struct for orchestrator authentication

* Adds function to whole logfile to Splunk

* Struct for pipeline related telemetry information

* Increase messagebatch size to 10k

* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk

* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation

* Changes content type to txt

* Send telemetry independent of logfiles, increases amount of messages per file

* Adds JobURL for orchestrators and UnknownOrchestrator as fallback

* telemetry makes use of orchestrator specific information

* Adds orchestrator independent correlationID

* Adds custom fields for pipeline status

* go fmt

* Removes env var test - no env variables are read anymore

* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized

* Removes Custom fields from telemetry as these can not be reflected in SWA

* Adds custom telemetry information (piperHash,..) to each step telemetry information

* Removes falltrough in case no orchestrator has been found

* Updates tests for orchestrator package

* Adds orchestrator import in generated files

* Updates generator files for internal library

* Adds orchestrator telemetry information to steps

* Updates generated files, fatalHook writes to cpe

* Go generate from master, go fmt

* Adds Custom Data field LastErrorCode

* Removes GetLog() test

* Update init_unix.go

* Update docker_integration_test_executor.go

* Update integration_api_cli_test.go

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Renames customTelemetryData to stepTelemetryData

* Adjustments to orchestrator-package, cleanup, adds JobName

* Adjusts commonPipelineEnvironment path

* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package

* Go fmt

* Changes path for errorDetails, adds debug information

* Removes custom fields from step, adds orchestrator, commithash to baseMetadata

* Adjusts tests for telemetry package

* Adds tests for orchestrator

* Updates generated files, initalization of splunk client only if its available in the config

* Fixes typo in helper go

* Update pkg/http/downloader.go

* Update pkg/http/downloader.go

* Update pkg/log/fatalHook.go

* Update fatalHook.go

* Update pkg/splunk/splunk.go

* Update pkg/telemetry/data.go

* Adds GetBuildStatus() and GetAPIInformation() to orchestrators

* error formatting

* Bugfix: dont send telemetry data if disabled, adjusts test

* go fmt

* Use correct error handling

* Update pkg/telemetry/telemetry.go

* Fixes telemetry disabled in the tests

* Fixes http tests

* Log fatal errors to logFile

* Adds CustomReportingConfig to hooks

* Cleanup comments in splunk package

* Adds possibility to send telemetry to custom endpoint

* Adds debug output for the payload

* Debug output for the payload as a string

* Adds test cases for changes in telemetry package

* go fmt

* Adds generated files for new step

* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17

* Adjusts test for sonarExecuteScan

* Adjusts test for sonarExecuteScan

* Adds explanation for customreportingConfig

* Makes disableing of customSend more obvious

* Adds custom step reporting to each step, updates generated files, adjusts helper testdata

* fixes unit test wrong usage of logging

* Send pipeline data altough there has been no error, adjust test cases

* Reverts changes for customReporting

* Updates generated files, removes customReporting

* Removes writing errorDetails to CPE

* Reverts usage of customreporting

* go fmt

* reverts changes in http_test

* reverts changes in http_test

* Skips integration cnb test

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 17:50:03 +01:00
Mayur Belur Mohan
d3b090e63e
ApiKeyValueMapDownload Command (#3245)
* ApiKeyValueMapDownload Command

* CodeReview Fixes
2021-11-15 13:48:14 +01:00
Mayur Belur Mohan
3ee4339af1
ApiProxyDownload Command (#3197)
* ApiProxyDownload Command

* Lint Fixes

* Lint Fixes

* codereview fixes

* Code Review Fixes

* CodeReview Fixes

* CodeReview Fixes

* Code Review Fixes

* Code Review Changes

* CodeReview Fixes

* CodeReview Fixes

* CodeReview Fix

* CodeReview Fixes

* CodeReviw Fixes

* CodeReview Changes

* CodeReview Fixes

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2021-11-02 10:30:08 +01:00
Roland Stengel
7612272992
checkChangeInDevelopment (#3115)
- refactor GO implementation
2021-10-04 13:08:34 +02:00
Christopher Fenner
4525c1daa4
fix: use orchestrator specific stage name (#3127)
* extend orchestator to provide stage name

* use orchestrator specific stage name

* fix test case

* remove comment

* fix test case

* prettify

* change something..

* do not exit

* Update pkg/orchestrator/azureDevOps.go

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-09-29 08:31:45 +02:00
Dominik Lendle
09f5c0a524
AUnit step (#3075)
* Add AUnit step

* Add Groovy File

* Update metadata

* Add groovy test config

* Add Unit tests

* Add Documentation

* Docu fixes

* change docu

* change docu

* change logging level & remove sleep

* change docu

* change docu

* Update documentation/docs/steps/abapEnvironmentRunAUnitTest.md

Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
2021-09-20 11:14:13 +02:00
Haegi
3f4b32f7ba
cnbBuild step MVP (#3021)
* Implement cnbBuild step

Co-authored-by: Benjamin Haegenlaeuer <benjamin.haegenlaeuer@sap.com>

* Add cnbBuild groovy test

Co-authored-by: Benjamin Haegenlaeuer <benjamin.haegenlaeuer@sap.com>

* Add basic documentation template

Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>

* Support specifiying name, tag and registry

Co-authored-by: Pavel Busko <pbusko@users.noreply.github.com>

Co-authored-by: Johannes Dillmann <j.dillmann@sap.com>
Co-authored-by: Philipp Stehle <philipp.stehle@sap.com>
Co-authored-by: Pavel Busko <pbusko@users.noreply.github.com>
2021-08-18 12:10:55 +02:00
Oliver Nocon
481f0a1282
fix(getConfig): custom metadata (#3047)
* fix(getConfig): custom metadata

when re-using getConfig it is now possible to provide a custom metadata resolver

* clean up coding
2021-08-12 14:54:46 +02:00
Mayur Belur Mohan
2d412d9f3c
IntegrationArtifactResource Command (#3027)
* IntegrationArtifactResource Command

* Remove unused code

* Formatting fix

* formatting fix

* formatting fix

* formatting fix

* formatting fix

* CodeReview Fixes

* Code Review Fix

* Code Review Fixes

* Code Review Fixes

* Format fix

* format fix

* format fix

* format fix

* Code Review Fix

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2021-08-12 09:11:02 +02:00
Mayur Belur Mohan
c283b9319d
IntegrationArtifactUnDeploy Command (#3018)
* IntegrationArtifactUnDeploy Command

* formatting fix

* Code Review Fixex

* Code Review Fixes

* remove unused code

* Formating fixes

* formatting fixes

* formatting fix

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
2021-08-02 16:27:16 +02:00
Siarhei Pazdniakou
7a325e6fc8
Convert step/stage condition logic to golang (#2993)
* Added checkIfStepActive step

* Implemented npmScripts condition. Code was refactored

* Added some unit tests

* Fixed go modules

* Fixed go modules

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-07-26 07:47:43 +02:00
Oliver Nocon
8458dd6afa
chore: export GitHub token unmarshalling (#2988) 2021-07-16 08:59:53 +02:00
Oliver Nocon
805a8fd88f
feat(config): read config/defaults with authentication (#2975)
* feat(config):read config/defaults with authentication

This change allows to use defaults and config files from a protected GitHub repository.

The options `--customConfig` and `--defaultConfig` already allowed to provide a link to an uprotected file.

Now, by passing a value in the form `<hostname>:<token>` to parameter `gitHubTokens` (this parameter can be passed multiple times) a token can be provided for dedicated hosts.

This makes it possible to use a link like
`https://api.github.com/repos/SAP/jenkins-library/contents/resources/my-defaults.yml?ref=master`
as reference to a default file or similarly as reference to a configuration file.

* update generation to allow protected config/defaults

* fix CodeClimate issues

* update missing generations
2021-07-08 15:26:07 +02:00
Siarhei Pazdniakou
54f2a0d471
Added go-based influxWriteData step (#2890)
* Added go-based influxWriteData step

* Wrote tests & fixed issues

* Fixed issues

* Created go-based step tests. Fixed issues

* Fixed issues

* Integration test was added

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2021-06-30 11:18:49 +02:00
Thorsten Duda
7910df0e8c
new step integrationArtifactTriggerIntegrationTest (#2951)
* new step integrationArtifactTriggerIntegrationTest

* add new step into allow list

* add the new step to main command

* refer cpe

* remove unused unit tests

* Check methods and URLs of http request

* Add TriggerIntegration to mockingutils

* Format code

Co-authored-by: Oliver Feldmann <oliver.feldmann@sap.com>
Co-authored-by: Linda Siebert <linda.siebert@sap.com>
2021-06-29 14:50:19 +02:00
Marcus Holl
fe5ce61d9e
rfc upload (#2533)
* Add RFC upload command
2021-06-14 12:36:18 +02:00
Kevin Stiehl
6c7814e4d5
feat(cpm): Add read and write CPE Go step (#2888)
* add read write cpe go steps

* Update pkg/piperenv/CPEMap.go

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* Update pkg/piperenv/CPEMap.go

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* Update pkg/piperenv/environment.go

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* rename file

* add error handling

* add error handling

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2021-06-11 16:17:40 +02:00
Mikalai Dzemidzenka
c38d231820
feat(gauge): migrate gaugeExecuteTests to go implementation (#2775)
* gaugeExecuteTests converted to golang

* rewrited gaugeExecuteTests to cross-platform implementation. Now gauge uses npm

* regenerated

* groovy file import fix

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-06-01 13:15:10 +02:00
Roland Stengel
02dd60c0c8
Split git commit message scan from solman upload (#2773)
* Split git commit message scan from solman upload
2021-05-21 08:54:43 +02:00
ffeldmann
b88ebdad6c
feat(splunk) Sending telemetry and logging information to Splunk (#2796)
* Adds inital splunk hook and logCollector

* Adds documentation of the Splunk hook

* Fixes markdown lint issues and removes comment from telemetry.go file

* Fixes markdown lint issues and adds missing generated file

* Markdown linting

* Changes documentation according to review, adds Splunk token automatically during init

* Adds error handling for marshalling hook config

* Markdown lint und correct Splunk token in httpclient

* Registeres Splunk token as secret and adjusts test cases

* Adds missing error handling and removes unnecessary comments

* Creates new function readPipelineEnvironment, adds tests

* Moves MonitoringData struct, edits defaults for json fields

* Adds gitRepository and gitOwner to telemetry information

* Simplifies readCommonPipelineEnvironment function, adds more descriptive errors, adds automated adding of Splunk prefix token

* Adjusts error handling

* Cleaner error logging
2021-05-17 12:14:04 +02:00
Mikalai Dzemidzenka
b82ecb0ff7
convert batsExecuteTests to go implementation (#2737)
* convert batsExecuteTests to go implementation

* added additional test cases, added container definition to batsExecuteTests.yaml

* added influx, for junit added container definition

* added parameter envVars

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-04-29 16:50:23 +02:00
Oliver Nocon
d04a7c2eb3
feat(pipelineCreateScanSummary) Create groovy wrapper (#2743)
* feat(pipelineCreateScanSummary) Create groovy wrapper

* add command to binary

* stash step reports

* update stash

* fix typo

* unstash reports first

* update reporting

* update json reporting

* update tests & enhance logging

* update md report

* update md reporting

* fix rendering

* update tests
2021-04-15 07:45:06 +02:00
Kevin Stiehl
fe6ea643fa
Add terraformExecuteStep (#2679)
* Add terraformExectueStep

* Update terraformExecute.go

* fmt

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-03-29 16:22:23 +02:00
Siarhei Pazdniakou
33699c7388
Convert containerExecuteStructureTests to go implementation (#2701)
* Converted containerExecuteStructureTests to go implementation

* Added tests

* Fixed issues

* Made fixes

Co-authored-by: lndrschlz <leander.schulz01@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-03-25 08:18:30 +01:00
Oliver Nocon
c038a3b08a
fix(whitesourceExecuteScan) properly pass verbose setting (#2713)
* fix(whitesourceExecuteScan) properly pass verbose setting

* check within piper binary

Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2021-03-23 12:06:23 +01:00