1
0
mirror of https://github.com/SAP/jenkins-library.git synced 2024-12-14 11:03:09 +02:00
Commit Graph

8 Commits

Author SHA1 Message Date
ffeldmann
d9e30418bc
feat: pipeline reporting, adds logging information to fatal hook and splunk hook (#3308)
* Removes unecessary fields from telemetry, restructuring splunk pkg

* Removes t.skip() and reactivates integration test

* Adjusts tests for fatalHook and helper functions, including log test

* Moves pipelinetelemetry to inner source, removes pipelineTelemetry from telemetry pkg, using generic map[string]interface for splunk

* Removes Read JSON from fatalHook -> moves to inner source

* Removed log output test

* go fmt

* log step telemetry data send to splunk

* Adjusts error logging

* Adds log information in case api information could not be retrieved

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-26 16:11:44 +01:00
ffeldmann
42b92d1bfe
Changes for Pipeline Reporting (#3213)
* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetLog() function to orchestrator

* Fixes BUILD_NUMBER env variable

* Fixes correct env var for JENKINS_HOME

* Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation

* Adds possibility to read errorJsons; updates splunk package for log files (WIP)

* Uncommenting dev code

* Adds GetRequest function which holds the response in memory (not saved to disk)

* Implements GetLog() function for ADO, adds function to read PipelineRuntime

* PAT has been revoked

* Changes http package, s.t. if password only is required basic auth works too

* Adds env variable for azure token, error handling in case of unauthenticated/nil response

* Adds logging output in case env variable can not be read and fallback variable needs to be used

* Adds usage of environment variables for auth, uses jenkins api

* Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function

* Adds initaliziation function for orchestrator authetnication

* Adds settings struct for orchestrator authentication

* Adds function to whole logfile to Splunk

* Struct for pipeline related telemetry information

* Increase messagebatch size to 10k

* Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk

* Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation

* Changes content type to txt

* Send telemetry independent of logfiles, increases amount of messages per file

* Adds JobURL for orchestrators and UnknownOrchestrator as fallback

* telemetry makes use of orchestrator specific information

* Adds orchestrator independent correlationID

* Adds custom fields for pipeline status

* go fmt

* Removes env var test - no env variables are read anymore

* Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized

* Removes Custom fields from telemetry as these can not be reflected in SWA

* Adds custom telemetry information (piperHash,..) to each step telemetry information

* Removes falltrough in case no orchestrator has been found

* Updates tests for orchestrator package

* Adds orchestrator import in generated files

* Updates generator files for internal library

* Adds orchestrator telemetry information to steps

* Updates generated files, fatalHook writes to cpe

* Go generate from master, go fmt

* Adds Custom Data field LastErrorCode

* Removes GetLog() test

* Update init_unix.go

* Update docker_integration_test_executor.go

* Update integration_api_cli_test.go

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Reverts go1.17 fmt formatting

* Renames customTelemetryData to stepTelemetryData

* Adjustments to orchestrator-package, cleanup, adds JobName

* Adjusts commonPipelineEnvironment path

* Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package

* Go fmt

* Changes path for errorDetails, adds debug information

* Removes custom fields from step, adds orchestrator, commithash to baseMetadata

* Adjusts tests for telemetry package

* Adds tests for orchestrator

* Updates generated files, initalization of splunk client only if its available in the config

* Fixes typo in helper go

* Update pkg/http/downloader.go

* Update pkg/http/downloader.go

* Update pkg/log/fatalHook.go

* Update fatalHook.go

* Update pkg/splunk/splunk.go

* Update pkg/telemetry/data.go

* Adds GetBuildStatus() and GetAPIInformation() to orchestrators

* error formatting

* Bugfix: dont send telemetry data if disabled, adjusts test

* go fmt

* Use correct error handling

* Update pkg/telemetry/telemetry.go

* Fixes telemetry disabled in the tests

* Fixes http tests

* Log fatal errors to logFile

* Adds CustomReportingConfig to hooks

* Cleanup comments in splunk package

* Adds possibility to send telemetry to custom endpoint

* Adds debug output for the payload

* Debug output for the payload as a string

* Adds test cases for changes in telemetry package

* go fmt

* Adds generated files for new step

* Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17

* Adjusts test for sonarExecuteScan

* Adjusts test for sonarExecuteScan

* Adds explanation for customreportingConfig

* Makes disableing of customSend more obvious

* Adds custom step reporting to each step, updates generated files, adjusts helper testdata

* fixes unit test wrong usage of logging

* Send pipeline data altough there has been no error, adjust test cases

* Reverts changes for customReporting

* Updates generated files, removes customReporting

* Removes writing errorDetails to CPE

* Reverts usage of customreporting

* go fmt

* reverts changes in http_test

* reverts changes in http_test

* Skips integration cnb test

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 17:50:03 +01:00
Sven Merk
7fa31ae9cc
feat(http): Resilience via retry on intermittent communication issues (#2877)
* feat(http): Resilience on connectivity issues

* Update whitesourceExecuteScan.go

* Update splunk_test.go

* Fix initialization

* Change http interaction

* Fix fmt and tests

* Final test fix
2021-06-15 11:13:24 +02:00
ffeldmann
b88ebdad6c
feat(splunk) Sending telemetry and logging information to Splunk (#2796)
* Adds inital splunk hook and logCollector

* Adds documentation of the Splunk hook

* Fixes markdown lint issues and removes comment from telemetry.go file

* Fixes markdown lint issues and adds missing generated file

* Markdown linting

* Changes documentation according to review, adds Splunk token automatically during init

* Adds error handling for marshalling hook config

* Markdown lint und correct Splunk token in httpclient

* Registeres Splunk token as secret and adjusts test cases

* Adds missing error handling and removes unnecessary comments

* Creates new function readPipelineEnvironment, adds tests

* Moves MonitoringData struct, edits defaults for json fields

* Adds gitRepository and gitOwner to telemetry information

* Simplifies readCommonPipelineEnvironment function, adds more descriptive errors, adds automated adding of Splunk prefix token

* Adjusts error handling

* Cleaner error logging
2021-05-17 12:14:04 +02:00
Oliver Nocon
19c1732826
Telemetry: report error category (#2085) 2020-09-29 13:49:40 +02:00
Stephan Aßmus
804bd8e3e0
http.go: Set fine-grained timeouts (#1257)
* Replace the default maximum request deadline with a default timeout on the transport level.
* Keep the possibility to set a maximum request deadline.
2020-03-23 15:02:22 +01:00
Oliver Nocon
b542d15998
Enhance telemetry reporting (#1133)
* Enhance telemetry reporting
* Use central telemetry data object
* Add duration

Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-01-31 14:06:08 +01:00
Christopher Fenner
aa3fb8adb4
feat(go): add telemetry reporting (#1100)
* Add telemetry support

* First round telemetry

* Add telemetry flag

* fix: move files to avoid import cycles

* add noTelemetry as global config option

* Respect telemetry configuration for reporting

* add site id, swa endpoint

* correct logger initialization

* add http logic

* rename init method

* rename consts & types

* convert struct to payload

* convert data to payload string

* move activation flag out of data structure

* extract types to own file

* build query using net/url

* correct field mapping

* extract notify coding to own file

* cleanup parameter mapping

* preare base data

* fix codeclimate issue

* correct test case

* fill values from env

* test all fields

* untrack notify.go

* ignore empty custom values

* cleanup data.go

* add test cases

* cleanup

* add usage reporting to karma step

* add usage reporting to step generator

* externalise siteID

* correct custom field names

* test env handling

* simplify method signature

* revert parameter negation

* correct import

* adjust golden file

* inclease log level

* ignore test case

* Revert "inclease log level"

This reverts commit 70cae0e029.

* add test case for envvars

* remove duplicate reporting

* remove duplicate reporting

* correct format

* regenerate checkmarx file

* add log message on deactivation

* rename function

* add comments to understand SWA mapping

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 13:17:54 +01:00