jenkins/sap-jenkins-library
1
0
Fork 0
mirror of https://github.com/SAP/jenkins-library.git synced 2024-12-14 11:03:09 +02:00
Code Issues Releases Activity
5926aa7f77
sap-jenkins-library/pkg/telemetry/data.go

113 lines
4.3 KiB
Go
Raw Normal View History

feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
package telemetry
import (
"encoding/json"
"net/url"
)
// BaseData object definition containing the base data and it's mapping information
type BaseData struct {
// SWA receives the fields custom1 - custom30 and e_a, e_2 - e_30 for custom values.
ActionName string `json:"action_name"`
EventType string `json:"event_type"`
SiteID string `json:"idsite"`
URL string `json:"url"`
StepName string `json:"e_3"` // set by step generator
StageName string `json:"e_10"`
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
PipelineURLHash string `json:"e_4"` // defaults to sha1 of provider.GetBuildUrl()
BuildURLHash string `json:"e_5"` // defaults to sha1 of provider.GetJobUrl()
Orchestrator string `json:"e_14"` // defaults to provider.OrchestratorType()
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
}
var baseData BaseData
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
// BaseMetaData object definition containing the labels for the base data, and it's mapping information
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
type BaseMetaData struct {
// SWA receives the fields custom1 - custom30 and e_a, e_2 - e_30 for custom values.
StepNameLabel string `json:"custom3"`
StageNameLabel string `json:"custom10"`
PipelineURLHashLabel string `json:"custom4"`
BuildURLHashLabel string `json:"custom5"`
Enhance telemetry reporting (#1133) * Enhance telemetry reporting * Use central telemetry data object * Add duration Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-01-31 15:06:08 +02:00
DurationLabel string `json:"custom11,omitempty"`
ExitCodeLabel string `json:"custom12,omitempty"`
Telemetry: report error category (#2085)
2020-09-29 13:49:40 +02:00
ErrorCategoryLabel string `json:"custom13,omitempty"`
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
OrchestratorLabel string `json:"custom14,omitempty"`
PiperCommitHashLabel string `json:"custom15,omitempty"`
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
}
// baseMetaData object containing the labels for the base data
var baseMetaData BaseMetaData = BaseMetaData{
StepNameLabel: "stepName",
StageNameLabel: "stageName",
PipelineURLHashLabel: "pipelineUrlHash",
BuildURLHashLabel: "buildUrlHash",
Enhance telemetry reporting (#1133) * Enhance telemetry reporting * Use central telemetry data object * Add duration Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-01-31 15:06:08 +02:00
DurationLabel: "duration",
ExitCodeLabel: "exitCode",
Telemetry: report error category (#2085)
2020-09-29 13:49:40 +02:00
ErrorCategoryLabel: "errorCategory",
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
OrchestratorLabel: "orchestrator",
PiperCommitHashLabel: "piperCommitHash",
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
}
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
// CustomData object definition containing the data that can be set by a step, and it's mapping information
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
type CustomData struct {
// SWA receives the fields custom1 - custom30 and e_a, e_2 - e_30 for custom values.
// Piper uses the values custom11 - custom25 & e_11 - e_25 for library related reporting
// and custom26 - custom30 & e_26 - e_30 for step related reporting.
Changes for Pipeline Reporting (#3213) * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetLog() function to orchestrator * Fixes BUILD_NUMBER env variable * Fixes correct env var for JENKINS_HOME * Adds getEnv to read env variables with default value, adds test for jenkins GetLog() implementation * Adds possibility to read errorJsons; updates splunk package for log files (WIP) * Uncommenting dev code * Adds GetRequest function which holds the response in memory (not saved to disk) * Implements GetLog() function for ADO, adds function to read PipelineRuntime * PAT has been revoked * Changes http package, s.t. if password only is required basic auth works too * Adds env variable for azure token, error handling in case of unauthenticated/nil response * Adds logging output in case env variable can not be read and fallback variable needs to be used * Adds usage of environment variables for auth, uses jenkins api * Adds init functionality for orchestrators, updates GetLog() and GetPipelineStartTime() function * Adds initaliziation function for orchestrator authetnication * Adds settings struct for orchestrator authentication * Adds function to whole logfile to Splunk * Struct for pipeline related telemetry information * Increase messagebatch size to 10k * Changes splunk package to a pointer based implementation, updates generated files and corresponding template and tests for splunk * Changes telemetry package to pointer based implementation to have multiple telemetry objects, adjusted tests and splunk implementation * Changes content type to txt * Send telemetry independent of logfiles, increases amount of messages per file * Adds JobURL for orchestrators and UnknownOrchestrator as fallback * telemetry makes use of orchestrator specific information * Adds orchestrator independent correlationID * Adds custom fields for pipeline status * go fmt * Removes env var test - no env variables are read anymore * Use UnknownOrchestratorConfigProvider in case the orchestrator can not be initalized * Removes Custom fields from telemetry as these can not be reflected in SWA * Adds custom telemetry information (piperHash,..) to each step telemetry information * Removes falltrough in case no orchestrator has been found * Updates tests for orchestrator package * Adds orchestrator import in generated files * Updates generator files for internal library * Adds orchestrator telemetry information to steps * Updates generated files, fatalHook writes to cpe * Go generate from master, go fmt * Adds Custom Data field LastErrorCode * Removes GetLog() test * Update init_unix.go * Update docker_integration_test_executor.go * Update integration_api_cli_test.go * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Reverts go1.17 fmt formatting * Renames customTelemetryData to stepTelemetryData * Adjustments to orchestrator-package, cleanup, adds JobName * Adjusts commonPipelineEnvironment path * Adds pipelineTelemetry struct to telemetry package, removes pipeline telemetry structs from splunk package * Go fmt * Changes path for errorDetails, adds debug information * Removes custom fields from step, adds orchestrator, commithash to baseMetadata * Adjusts tests for telemetry package * Adds tests for orchestrator * Updates generated files, initalization of splunk client only if its available in the config * Fixes typo in helper go * Update pkg/http/downloader.go * Update pkg/http/downloader.go * Update pkg/log/fatalHook.go * Update fatalHook.go * Update pkg/splunk/splunk.go * Update pkg/telemetry/data.go * Adds GetBuildStatus() and GetAPIInformation() to orchestrators * error formatting * Bugfix: dont send telemetry data if disabled, adjusts test * go fmt * Use correct error handling * Update pkg/telemetry/telemetry.go * Fixes telemetry disabled in the tests * Fixes http tests * Log fatal errors to logFile * Adds CustomReportingConfig to hooks * Cleanup comments in splunk package * Adds possibility to send telemetry to custom endpoint * Adds debug output for the payload * Debug output for the payload as a string * Adds test cases for changes in telemetry package * go fmt * Adds generated files for new step * Reverts changes for http tests, causing problems with go1.15, changes need to be applied for newer go version >=1.17 * Adjusts test for sonarExecuteScan * Adjusts test for sonarExecuteScan * Adds explanation for customreportingConfig * Makes disableing of customSend more obvious * Adds custom step reporting to each step, updates generated files, adjusts helper testdata * fixes unit test wrong usage of logging * Send pipeline data altough there has been no error, adjust test cases * Reverts changes for customReporting * Updates generated files, removes customReporting * Removes writing errorDetails to CPE * Reverts usage of customreporting * go fmt * reverts changes in http_test * reverts changes in http_test * Skips integration cnb test Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2021-11-18 18:50:03 +02:00
Duration string `json:"e_11,omitempty"`
ErrorCode string `json:"e_12,omitempty"`
ErrorCategory string `json:"e_13,omitempty"`
PiperCommitHash string `json:"e_15,omitempty"`
Custom1Label string `json:"custom26,omitempty"`
Custom2Label string `json:"custom27,omitempty"`
Custom3Label string `json:"custom28,omitempty"`
Custom4Label string `json:"custom29,omitempty"`
Custom5Label string `json:"custom30,omitempty"`
Custom1 string `json:"e_26,omitempty"`
Custom2 string `json:"e_27,omitempty"`
Custom3 string `json:"e_28,omitempty"`
Custom4 string `json:"e_29,omitempty"`
Custom5 string `json:"e_30,omitempty"`
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
}
Reporting: Variable change to deprecate step Splunk Hook; logging in telemetry (#3539) * Adds fatalError variable with helper setter and getter helper functions * Adds logging function to telemetry package (always executed)
2022-02-14 09:56:11 +02:00
// StepTelemetryData definition for telemetry reporting and monitoring
type StepTelemetryData struct {
Updates telemetry logging information for internal reporting (#3585) * Add StepStartTime, Renames StepDuration, adds PiperCommithash, removes Branch, GitOwner, GitRepository from logged telemetry information * Fixes test case for telemetry logging * Activates step monitoring data in debug mode * Pretty debug json printing * Reduces log noise, setting warning to debug
2022-02-28 10:45:57 +02:00
StepStartTime string `json:"StepStartTime"`
Reporting: Variable change to deprecate step Splunk Hook; logging in telemetry (#3539) * Adds fatalError variable with helper setter and getter helper functions * Adds logging function to telemetry package (always executed)
2022-02-14 09:56:11 +02:00
PipelineURLHash string `json:"PipelineURLHash"`
BuildURLHash string `json:"BuildURLHash"`
StageName string `json:"StageName"`
StepName string `json:"StepName"`
ErrorCode string `json:"ErrorCode"`
Updates telemetry logging information for internal reporting (#3585) * Add StepStartTime, Renames StepDuration, adds PiperCommithash, removes Branch, GitOwner, GitRepository from logged telemetry information * Fixes test case for telemetry logging * Activates step monitoring data in debug mode * Pretty debug json printing * Reduces log noise, setting warning to debug
2022-02-28 10:45:57 +02:00
StepDuration string `json:"StepDuration"`
Reporting: Variable change to deprecate step Splunk Hook; logging in telemetry (#3539) * Adds fatalError variable with helper setter and getter helper functions * Adds logging function to telemetry package (always executed)
2022-02-14 09:56:11 +02:00
ErrorCategory string `json:"ErrorCategory"`
CorrelationID string `json:"CorrelationID"`
Updates telemetry logging information for internal reporting (#3585) * Add StepStartTime, Renames StepDuration, adds PiperCommithash, removes Branch, GitOwner, GitRepository from logged telemetry information * Fixes test case for telemetry logging * Activates step monitoring data in debug mode * Pretty debug json printing * Reduces log noise, setting warning to debug
2022-02-28 10:45:57 +02:00
PiperCommitHash string `json:"PiperCommitHash"`
Reporting: Variable change to deprecate step Splunk Hook; logging in telemetry (#3539) * Adds fatalError variable with helper setter and getter helper functions * Adds logging function to telemetry package (always executed)
2022-02-14 09:56:11 +02:00
ErrorDetail map[string]interface{} `json:"ErrorDetail"`
}
feat(go): add telemetry reporting (#1100) * Add telemetry support * First round telemetry * Add telemetry flag * fix: move files to avoid import cycles * add noTelemetry as global config option * Respect telemetry configuration for reporting * add site id, swa endpoint * correct logger initialization * add http logic * rename init method * rename consts & types * convert struct to payload * convert data to payload string * move activation flag out of data structure * extract types to own file * build query using net/url * correct field mapping * extract notify coding to own file * cleanup parameter mapping * preare base data * fix codeclimate issue * correct test case * fill values from env * test all fields * untrack notify.go * ignore empty custom values * cleanup data.go * add test cases * cleanup * add usage reporting to karma step * add usage reporting to step generator * externalise siteID * correct custom field names * test env handling * simplify method signature * revert parameter negation * correct import * adjust golden file * inclease log level * ignore test case * Revert "inclease log level" This reverts commit 70cae0e0296afb2aa9e7d71e83ea70aa83d1a6d7. * add test case for envvars * remove duplicate reporting * remove duplicate reporting * correct format * regenerate checkmarx file * add log message on deactivation * rename function * add comments to understand SWA mapping Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 14:17:54 +02:00
// Data object definition containing all telemetry data
type Data struct {
BaseData
BaseMetaData
CustomData
}
// toMap transfers the data object into a map using JSON tags
func (d *Data) toMap() (result map[string]string) {
jsonObj, _ := json.Marshal(d)
json.Unmarshal(jsonObj, &result)
return
}
// toPayloadString transfers the data object into a 'key=value&..' string
func (d *Data) toPayloadString() string {
parameters := url.Values{}
for key, value := range d.toMap() {
if len(value) > 0 {
parameters.Add(key, value)
}
}
return parameters.Encode()
}
Reference in New Issue Copy Permalink