1
0
mirror of https://github.com/SAP/jenkins-library.git synced 2024-12-14 11:03:09 +02:00
Commit Graph

45 Commits

Author SHA1 Message Date
ffeldmann
ccc1c976ee
fix(orchestrator) usage of correct env variables (#3650)
* Reorders getApiInformation, changes variables to get start time, adjusts and adds test cases
* Changes the way to get apiInformation and reduces number of requests
* Changes getting pipeline start time from correct env variable
* Refactors getApiInformation functionality
* Adds GetBuildReason() for Azure and Jenkins
* Updates JobURL for ADO
2022-03-28 09:52:15 +02:00
Christian Volk
22f6aa156f
feat(docker): use crane for pulling docker images (#3652) 2022-03-23 10:02:00 +01:00
Oliver Nocon
0998a80f07
chore: update to go 1.17 (#3557)
* chore: update to go 1.17

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>

* update workflows

Co-authored-by: Pavel Busko <pavel.busko@sap.com>

* fixed multipart upload tests

Co-authored-by: Johannes Dillmann <j.dillmann@sap.com>

* go fmt

Co-authored-by: Pavel Busko <pavel.busko@sap.com>

* Update to go 1.17

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>

* Update vault to v1.9.3

Co-authored-by: Pavel Busko <pavel.busko@sap.com>

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
Co-authored-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Johannes Dillmann <j.dillmann@sap.com>
Co-authored-by: ffeldmann <f.feldmann@sap.com>
2022-03-07 17:15:36 +01:00
Sven Merk
a1988f6808
feat(whitesourceExecuteScan): GitHub issue creation + SARIF (#3535)
* Add GH issue creation + SARIF

* Code cleanup

* Fix fmt, add debug

* Code enhancements

* Fix

* Added debug info

* Rework UA log scan

* Fix code

* read UA version

* Fix nil reference

* Extraction

* Credentials

* Issue creation

* Error handling

* Fix issue creation

* query escape

* Query escape 2

* Revert

* Test avoid update

* HTTP client

* Add support for custom TLS certs

* Fix code

* Fix code 2

* Fix code 3

* Disable cert check

* Fix auth

* Remove implicit trust

* Skip verification

* Fix

* Fix client

* Fix HTTP auth

* Fix trusted certs

* Trim version

* Code

* Add token

* Added token handling to client

* Fix token

* Cleanup

* Fix token

* Token rework

* Fix code

* Kick out oauth client

* Kick out oauth client

* Transport wrapping

* Token

* Simplification

* Refactor

* Variation

* Check

* Fix

* Debug

* Switch client

* Variation

* Debug

* Switch to cert check

* Add debug

* Parse self

* Cleanup

* Update resources/metadata/whitesourceExecuteScan.yaml

* Add debug

* Expose subjects

* Patch

* Debug

* Debug2

* Debug3

* Fix logging response body

* Cleanup

* Cleanup

* Fix request body logging

* Cleanup import

* Fix import cycle

* Cleanup

* Fix fmt

* Fix NopCloser reference

* Regenerate

* Reintroduce

* Fix test

* Fix tests

* Correction

* Fix error

* Code fix

* Fix tests

* Add tests

* Fix code climate issues

* Code climate

* Code climate again

* Code climate again

* Fix fmt

* Fix fmt 2

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2022-02-23 09:30:19 +01:00
Christian Volk
54ca265860
fix(httpclient): folder permissions (#3506) 2022-02-04 14:40:18 +01:00
ffeldmann
42b92d1bfe
Changes for Pipeline Reporting (#3213)
* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetRequest function which holds the response in memory (not saved to disk)

* Implements GetLog() function for ADO, adds function to read PipelineRuntime

* PAT has been revoked

* Changes http package, s.t. if password only is required basic auth works too

* Adds env variable for azure token, error handling in case of unauthenticated/nil response

* Adds logging output in case env variable can not be read and fallback variable needs to be used

* Adds usage of environment variables for auth, uses jenkins api

* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function

* Adds initaliziation function for orchestrator authetnication

* Adds settings struct for orchestrator authentication

* Adds function to whole logfile to Splunk

* Struct for pipeline related telemetry information

* Increase messagebatch size to 10k

* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk

* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation

* Changes content type to txt

* Send telemetry independent of logfiles, increases amount of messages per file

* Adds JobURL for orchestrators and UnknownOrchestrator as fallback

* telemetry makes use of orchestrator specific information

* Adds orchestrator independent correlationID

* Adds custom fields for pipeline status

* go fmt

* Removes env var test - no env variables are read anymore

* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized

* Removes Custom fields from telemetry as these can not be reflected in SWA

* Adds custom telemetry information (piperHash,..) to each step telemetry information

* Removes falltrough in case no orchestrator has been found

* Updates tests for orchestrator package

* Adds orchestrator import in generated files

* Updates generator files for internal library

* Adds orchestrator telemetry information to steps

* Updates generated files, fatalHook writes to cpe

* Go generate from master, go fmt

* Adds Custom Data field LastErrorCode

* Removes GetLog() test

* Update init_unix.go

* Update docker_integration_test_executor.go

* Update integration_api_cli_test.go

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Renames customTelemetryData to stepTelemetryData

* Adjustments to orchestrator-package, cleanup, adds JobName

* Adjusts commonPipelineEnvironment path

* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package

* Go fmt

* Changes path for errorDetails, adds debug information

* Removes custom fields from step, adds orchestrator, commithash to baseMetadata

* Adjusts tests for telemetry package

* Adds tests for orchestrator

* Updates generated files, initalization of splunk client only if its available in the config

* Fixes typo in helper go

* Update pkg/http/downloader.go

* Update pkg/http/downloader.go

* Update pkg/log/fatalHook.go

* Update fatalHook.go

* Update pkg/splunk/splunk.go

* Update pkg/telemetry/data.go

* Adds GetBuildStatus() and GetAPIInformation() to orchestrators

* error formatting

* Bugfix: dont send telemetry data if disabled, adjusts test

* go fmt

* Use correct error handling

* Update pkg/telemetry/telemetry.go

* Fixes telemetry disabled in the tests

* Fixes http tests

* Log fatal errors to logFile

* Adds CustomReportingConfig to hooks

* Cleanup comments in splunk package

* Adds possibility to send telemetry to custom endpoint

* Adds debug output for the payload

* Debug output for the payload as a string

* Adds test cases for changes in telemetry package

* go fmt

* Adds generated files for new step

* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17

* Adjusts test for sonarExecuteScan

* Adjusts test for sonarExecuteScan

* Adds explanation for customreportingConfig

* Makes disableing of customSend more obvious

* Adds custom step reporting to each step, updates generated files, adjusts helper testdata

* fixes unit test wrong usage of logging

* Send pipeline data altough there has been no error, adjust test cases

* Reverts changes for customReporting

* Updates generated files, removes customReporting

* Removes writing errorDetails to CPE

* Reverts usage of customreporting

* go fmt

* reverts changes in http_test

* reverts changes in http_test

* Skips integration cnb test

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 17:50:03 +01:00
Umidjon Urunov
53225b958f
protecodeExecuteScan -> file upload has been extended for "binary" uploads (#3156)
* changes to detectExec before master merge

* changes for detectExecuteScan

* self generated code added

* fix syntax errors and update docu

* added unit tests for fail and Group

* fix failOn bug

* add Groups as string array

* add Groups as string array

* tests and validation for groups, failOn

* Updated docs and added more tests

* documentation md files should not be changed

* Handle merge conflicts from PR 1845

* fix merge errors

* remove duplicate groups, merge error

* adding buildCode and buildTool as params

* switching build options

* building maven modules

* parameter correction

* parameter correction

* gnerate with new build parameter

* adding comments

* removing piper lib master and modifying goUtils to download 1.5.7 release

* first cleaning then installing

* multi module maven built

* multi module maven built removing unwanted code

* multi module maven built moving inside switch

* testing

* modifying the default use case to also call maven build

* modifying the default use case to also call maven build wih --

* corrected maven build command

* corrected maven build command with %v

* skipping test runs

* testing for MTA project with single pom

* adding absolute path to m2 path

* clean up

* adding switch for mta and maven and removing env from containers

* commiting changes for new detect step

* correting log message

* code clean up

* unit tests changes to detectExecute

* basic tests for new change

* restoring piperGoUtils to download correct piper binary

* code clean up

* code clean up

* protecodeExecuteScan :: fixing file upload for binaries

* protecodeExecuteScan :: fixing protecode generate file

* Fix upload test

* protecodeExecuteScan -> fixing tests

Co-authored-by: D072410 <giridhar.shenoy@sap.com>
Co-authored-by: Keshav <anil.keshav@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2021-10-21 10:03:42 +02:00
Daniel Mieg
e9b3e79f3a
Change loglevel to debug (#3107) 2021-09-23 10:05:44 +02:00
Anil Keshav
d29ba346f2
[feat] https communication enhancement : Eliminate skip tls skip verify in the piper http client (#3062)
* modify wrapper http to hold ca rot

* modifying the http client for maven build

* adding checks

* eliminating seperate jks

* test tls true

* insecure flag remove

* error debug

* storing cert

* sap root cert

* error

* only child cert

* test

* maven test

* moving outside loop

* changing pointer

* dont download existing certs

* typo fix

* removing mavenBuild test

* code clean up

* making hadolint using always cert true

* custom tls link for hadolint trust

* error handel

* extended condition for modifying custom tls

* unit test case

* checing when to add the customLinks

* not breaking existing hadolint client config

Co-authored-by: Your Name <you@example.com>
2021-08-19 11:29:33 +02:00
Sven Merk
ab9e154d10
Replace io.Copy (#2934)
* Replace io.Copy

* Test coverage

* Improve test

* Fix fmt

* Improve error handling in test

* Fix code

* Improve test error log

* Fix fmt

* Fix unix file handles

* Fix error message

* Resolve code climate issue
2021-06-23 14:41:52 +02:00
Sven Merk
7fa31ae9cc
feat(http): Resilience via retry on intermittent communication issues (#2877)
* feat(http): Resilience on connectivity issues

* Update whitesourceExecuteScan.go

* Update splunk_test.go

* Fix initialization

* Change http interaction

* Fix fmt and tests

* Final test fix
2021-06-15 11:13:24 +02:00
Christopher Fenner
62836a64e7
feat(http): expose standard client (#2860)
* expose standard client

* reset go.summ
2021-05-28 12:13:19 +02:00
Sven Merk
ff909cb630
Update http.go (#2744) 2021-04-07 15:02:40 +02:00
Sven Merk
60fe1761b6
Update http.go (#2731)
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-03-31 16:08:35 +02:00
Sven Merk
e1ea56076f
Http improve retry on timeouts (#2681)
* Add sca cmd extensibility

* Fix formatting

* HTTP retry

* Improve handling of retry on timeout

* Go fmt

* Fix test

* Fix test

* Test stability

* Fix test

* Fix test

* Fix test

* Update fortifyExecuteScan.go
2021-03-09 13:41:07 +01:00
Christopher Fenner
5184a76453
chore(http): allow to send prepared requests (#2580)
* Update http.go

* fix c&p issue

* convert URL to string

* Apply suggestions from code review

fix code climate findings

* add test case
2021-02-08 14:26:15 +01:00
Christopher Fenner
9ec282fd0c
test: allow usage of httpmock in tests (#2576)
* add test case

* add flag for default transport

* Update go.mod

* Update go.sum

* fix spaces

* fix format

* Update http_test.go

* Update http_test.go

* Update http_test.go

* Update http_test.go

* Update http_test.go
2021-02-04 14:58:35 +01:00
Sven Merk
3e0a2835fd
Improve HTTP test to cover more retry capabilities (#2528)
* FF disable fulltextsearch

# Conflicts:
#	pkg/fortify/fortify.go
#	pkg/fortify/fortify_test.go

* Improve test on HTTP retry capabilities

* Improve HTTP retry test
2021-01-22 09:56:56 +01:00
Oliver Feldmann
ce9b958410
Document SendRequest error case (#2507)
* Document error case

* Improve docu text
2021-01-12 15:26:45 +01:00
Marcus Holl
9172fcae04
Don't expose auth header to log (#2334)
Do not expose auth header to log
2021-01-04 10:06:28 +01:00
Marcus Holl
f9fc642c0e
http client: tell us which resource was not found (#2410)
Up to now we get a message 'Requested resource could not be found' which is not very
helpul during troubleshooting based on the log. With this change we tell the reader
which resource could not be resolved.
2020-12-16 08:55:22 +01:00
Sven Merk
3c7712f2ee
Retry capabilities for HTTP requests + enablement for Checkmarx step (#2346) 2020-11-11 13:35:53 +01:00
Christopher Fenner
e859ed3375
feat(http): expose tls insecure flag (#2324)
* expose tls insecure flag

* fix tls disabling

* move debug message

* add test case
2020-11-09 11:47:03 +01:00
Christopher Fenner
b7afe8bbf5
feat(http): support empty passwords (#2206)
* add test case

* allow zero-length passwords

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-11-03 19:36:52 +01:00
Christopher Fenner
6999380ee3
chore(go): simplify code using gofmt -s (#2065) 2020-09-24 08:58:53 +02:00
Christopher Fenner
b219fb6514
fix(typo): found by misspell (#2064)
* fix typos in step yamls

* fix typos in go files

* regenerate step code

* fix typos in md files

* fix typos in groovy files

* fix further typos
2020-09-24 07:41:06 +02:00
Chris Bo
9558b47268
Introducing new step 'gctsExecuteABAPUnitTests' (#1532)
* add new step 'gctsExecuteABAPUnitTests'

* added new step entries

* small wording improvement

* regenerated

* fixed unit tests

* adjusted prerequisites

* removed telemetry

* remove command

* moved functions to http package

* newly generated

* remove commented code

* use credentials ID in config.yaml

* add supported protocols to docu

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-07-14 10:58:57 +02:00
Stephan Aßmus
c693c4c7dc
Increase default transport timeout to 3 mins (#1648) 2020-06-10 11:14:55 +02:00
Sven Merk
af2a01c064
Fortify implementation in golang (#1428) 2020-05-25 19:48:59 +02:00
Stephan Aßmus
dd94ba7213
TransportTimeout now refers to any state (#1467)
The TransportTimeout value is now applied to the "response header
timeout" and "expect continue timeout" as is. Previously there
was a hard limit of 10 seconds and 1 second respectively (originating
from the article I based the previous PR on). While this doesn't
allow for fine-grained control, it is reasonable to apply the meaning
of "transport timeout" to any state or phase of a HTTP connection.
This change should solve the needs of some Piper clients to configure
very long response header timeouts of 5 minutes and above.
2020-04-28 21:48:00 +02:00
Stephan Aßmus
527baf28fd
Future proof the http-timeout test for go 1.14 (#1321)
Fix flaky test expectations. The exact error message has changed in go 1.14.
2020-03-31 09:18:09 +02:00
Stephan Aßmus
804bd8e3e0
http.go: Set fine-grained timeouts (#1257)
* Replace the default maximum request deadline with a default timeout on the transport level.
* Keep the possibility to set a maximum request deadline.
2020-03-23 15:02:22 +01:00
Florian Wilhelm
fdb9fd3c6e
Print "Status" in error case in http client (#1256)
This might provide useful context to the user.

For example when you try to upload a SNAPSHOT jar to a maven-release
repo in nexus, this will print an error like

returned with response 400 Repository version policy: RELEASE does
not allow version: 1.0-SNAPSHOT

instead of

returned with HTTP Code 400
2020-03-09 19:25:40 +01:00
Marcus Holl
4835fbffab
Support nested folders when performing a http download into a file (#1206) 2020-02-25 16:52:12 +01:00
Marcus Holl
317716cf64
switch downloader from POST to GET (#1203) 2020-02-20 12:46:32 +01:00
Christopher Fenner
ea45136c3d
feat(go): add download file function (#1200)
* add download file function

* add test case

* Update pkg/piperutils/FileUtils.go

* correct test case

* remove FileUtils.Download

* add Downloader

* add Downloader

* fix error

* respect header and cookies

* add test case

* rename files

* correct test case

* remove SendRequest

* correct test case
2020-02-19 19:26:47 +01:00
redehnroV
2ebf2010b7
Protecode as GoLang (#1119)
* Protecode as go implementation

Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-02-06 16:16:34 +01:00
Daniel Mieg
6f0bde154b
Go implementation for abapEnvironmentPullGitRepo() (#1107) 2020-02-04 11:43:27 +01:00
Christopher Fenner
aa3fb8adb4
feat(go): add telemetry reporting (#1100)
* Add telemetry support

* First round telemetry

* Add telemetry flag

* fix: move files to avoid import cycles

* add noTelemetry as global config option

* Respect telemetry configuration for reporting

* add site id, swa endpoint

* correct logger initialization

* add http logic

* rename init method

* rename consts & types

* convert struct to payload

* convert data to payload string

* move activation flag out of data structure

* extract types to own file

* build query using net/url

* correct field mapping

* extract notify coding to own file

* cleanup parameter mapping

* preare base data

* fix codeclimate issue

* correct test case

* fill values from env

* test all fields

* untrack notify.go

* ignore empty custom values

* cleanup data.go

* add test cases

* cleanup

* add usage reporting to karma step

* add usage reporting to step generator

* externalise siteID

* correct custom field names

* test env handling

* simplify method signature

* revert parameter negation

* correct import

* adjust golden file

* inclease log level

* ignore test case

* Revert "inclease log level"

This reverts commit 70cae0e029.

* add test case for envvars

* remove duplicate reporting

* remove duplicate reporting

* correct format

* regenerate checkmarx file

* add log message on deactivation

* rename function

* add comments to understand SWA mapping

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 13:17:54 +01:00
Sven Merk
cbe368fe36
Checkmarx as golang (#1075)
* Added base functionality for checkmarx interaction

* Extend http client with file upload capabilities

* Latest changes

* Add debug logging

* Introduce Uploader interface

* Add tests for checkmarx client

* Hook new checkmarx command

* Improve coverage

* Add tests

* Improved test coverage and fixed code

* Add influx reporting

* Add alternation capabilities

* Add groovy step

* Try fix cmd

* Enhancements

* Fix report generation

* Final performance improvements

* Fix code

* Structure code, cleanup

* Improvements

* Fix codeclimate issue

* Update groovy

* Adapt latest changes to http

* Fix test

* Fix http tests

* Fix test

* Fix test

* Fix test 2

* Fix code

* Fix code 2

* Fix code

* Code

* Fix

* Fix

* Add report and link handling

* Fix returns, add groovy test

* Review comments

* Added doc template

* Docs update

* Remove SAP internals

* Better status display

* Add name to link

* Fix test

* Fix

* Fix verbose handling

* Fix verbose handling 2

* Fix verbose handling 3

* Fix

* Tiny improvements

* Regenerate

* Fix test

* Fix test code

* Fix verbosity issue

* Fix test

* Fix test

* Fix test
2020-01-27 23:40:53 +01:00
Sven Merk
71d0842c07
Update http_test.go 2020-01-22 15:26:35 +01:00
Sascha Vornheder
979ed1fabd added check that only POST and PUT is allowed for upload 2020-01-22 15:10:40 +01:00
Sascha Vornheder
1ae1be9cb9 improved Upload to handle http.Method 2020-01-22 14:22:04 +01:00
Sven Merk
acadf24d27 Add upload of files 2020-01-14 10:29:50 +01:00
Oliver Nocon
df1c61dcd7
Add http package (#1033)
* Add http package

This serves for http calls in future piper steps implemented ingolang
2019-12-09 17:35:31 +01:00