Add collection of library telemetry data (#239)

* add telemetry collection
* add telemetry reporting for first steps
* fix documentation formatting

documentation/docs/configuration.md

@@ -0,0 +1,83 @@
+# Configuration
+
+Configuration is done via a yml-file, located at `.pipeline/config.yml` in the **master branch** of your source code repository.
+
+Your configuration inherits from the default configuration located at [https://github.com/SAP/jenkins-library/blob/master/resources/default_pipeline_environment.yml](https://github.com/SAP/jenkins-library/blob/master/resources/default_pipeline_environment.yml).
+
+!!! caution "Adding custom parameters"
+    Please note that adding custom parameters to the configuration is at your own risk.
+    We may introduce new parameters at any time which may clash with your custom parameters.
+
+Configuration of the Piper steps as well the Piper templates can be done in a hierarchical manner.
+
+1. Directly passed step parameters will always take precedence over other configuration values and defaults
+2. Stage configuration parameters define a Jenkins pipeline stage dependent set of parameters (e.g. deployment options for the `Acceptance` stage)
+3. Step configuration defines how steps behave in general (e.g. step `cloudFoundryDeploy`)
+4. General configuration parameters define parameters which are available across step boundaries
+5. Default configuration comes with the Piper library and is always available
+
+![Piper Configuration](images/piper_config.png)
+
+## Collecting telemetry data
+
+In order to improve this Jenkins library we are collecting telemetry data.
+Data is send using [`com.sap.piper.pushToSWA`](https://github.com/SAP/jenkins-library/blob/master/src/com/sap/piper/Utils.groovy)
+
+Following data (non-personal) is collected for example:
+
+* Hashed job url, e.g. `4944f745e03f5f79daf0001eec9276ce351d3035` hash calculation is done in your Jenkins server and no original values are transmitted
+* Name of library step which has been executed, like e.g. `artifactSetVersion`
+* Certain parameters of the executed steps, e.g. `buildTool=maven`
+
+**We store the telemetry data for not longer than 6 months on premises of SAP SE.**
+
+!!! note "Disable collection of telemetry data"
+    If you do not want to send telemetry data which helps this open source project to improve you can easily deactivate this.
+
+    This is done with either of the following two ways:
+
+    1. General deactivation in your `.pipeline/config.yml` file by setting the configuration parameter `general -> collectTelemetryData: false` (default setting can be found in the [library defaults](https://github.com/SAP/jenkins-library/blob/master/resources/default_pipeline_environment.yml)).
+
+        **Please note: this will only take effect in all steps if you run `setupCommonPipelineEnvironment` at the beginning of your pipeline**
+
+    2. Individual deactivation per step by passing the parameter `collectTelemetryData: false`, like e.g. `setVersion script:this, collectTelemetryData: false`
+
+
+## Example configuration
+
+```
+general:
+  gitSshKeyCredentialsId: GitHub_Test_SSH
+
+steps:
+  cloudFoundryDeploy:
+    deployTool: 'cf_native'
+    cloudFoundry:
+      org: 'testOrg'
+      space: 'testSpace'
+      credentialsId: 'MY_CF_CREDENTIALSID_IN_JENKINS'
+  newmanExecute:
+    newmanCollection: 'myNewmanCollection.file'
+    newmanEnvironment: 'myNewmanEnvironment'
+    newmanGlobals: 'myNewmanGlobals'
+ ```
+
+## Access to configuration from custom scripts
+
+Configuration is loaded into `commonPipelineEnvironment` during step [setupCommonPipelineEnvironment](steps/setupCommonPipelineEnvironment.md).
+
+You can access the configuration values via `commonPipelineEnvironment.configuration` which will return you the complete configuration map.
+
+Thus following access is for example possible (accessing `gitSshKeyCredentialsId` from `general` section):
+```
+commonPipelineEnvironment.configuration.general.gitSshKeyCredentialsId
+```
+
+## Access to configuration in custom library steps
+
+Within library steps the `ConfigurationHelper` object is used.
+
+You can see its usage in all the Piper steps, for example [newmanExecute](https://github.com/SAP/jenkins-library/blob/master/vars/newmanExecute.groovy#L23).
+
+
+

documentation/mkdocs.yml

@@ -1,9 +1,10 @@
 site_name: Jenkins 2.0 Pipelines
 pages:
     - Home: index.md
+    - Configuration: configuration.md
     - 'Library steps':
         - artifactSetVersion: steps/artifactSetVersion.md
-        - checkChangeInDevelopment: steps/checkChangeInDevelopment.md   
+        - checkChangeInDevelopment: steps/checkChangeInDevelopment.md
         - commonPipelineEnvironment: steps/commonPipelineEnvironment.md
         - dockerExecute: steps/dockerExecute.md
         - durationMeasure: steps/durationMeasure.md

resources/default_pipeline_environment.yml

@@ -1,6 +1,7 @@
 #Project Setup
 general:
   productiveBranch: 'master'
+  collectTelemetryData: true
   changeManagement:
       transportRequestLabel: 'TransportRequest\s?:'
       changeDocumentLabel: 'ChangeDocument\s?:'

src/com/sap/piper/Utils.groovy

@@ -1,6 +1,7 @@
 package com.sap.piper
 
 import com.cloudbees.groovy.cps.NonCPS
+import org.jenkinsci.plugins.workflow.steps.MissingContextVariableException
 
 @NonCPS
 def getMandatoryParameter(Map map, paramName, defaultValue = null) {
@@ -51,3 +52,64 @@ def unstashAll(stashContent) {
     return unstashedContent
 }
 
+def generateSha1Inline(input) {
+    return "`echo -n '${input}' | sha1sum | sed 's/  -//'`"
+}
+
+void pushToSWA(Map parameters, Map config) {
+    try {
+        //allow opt-out via configuration
+        if (!config.collectTelemetryData) {
+            return
+        }
+
+        def swaCustom = [:]
+
+        /* SWA custom parameters:
+            custom3 = step name (passed as parameter step)
+            custom4 = job url hashed (calculated)
+            custom5 = build url hashed (calculated)
+            custom11 = step related parameter 1 (passed as parameter stepParam1)
+            custom12 = step related parameter 2 (passed as parameter stepParam2)
+            custom13 = step related parameter 3 (passed as parameter stepParam3)
+            custom14 = step related parameter 4 (passed as parameter stepParam4)
+            custom15 = step related parameter 5 (passed as parameter stepParam5)
+        */
+
+        def swaUrl = 'https://webanalytics.cfapps.eu10.hana.ondemand.com/tracker/log'
+        def action_name = 'Piper Library OS'
+        def idsite = '827e8025-1e21-ae84-c3a3-3f62b70b0130'
+        def url = 'https://github.com/SAP/jenkins-library'
+        def event_type = 'library-os'
+
+        swaCustom.custom3 = parameters.get('step')
+        swaCustom.custom4 = generateSha1Inline(env.JOB_URL)
+        swaCustom.custom5 = generateSha1Inline(env.BUILD_URL)
+        swaCustom.custom11 = parameters.get('stepParam1')
+        swaCustom.custom12 = parameters.get('stepParam2')
+        swaCustom.custom13 = parameters.get('stepParam3')
+        swaCustom.custom14 = parameters.get('stepParam4')
+        swaCustom.custom15 = parameters.get('stepParam5')
+
+        def options = []
+        options.push("-G")
+        options.push("-v \"${swaUrl}\"")
+        options.push("--data-urlencode \"action_name=${action_name}\"")
+        options.push("--data-urlencode \"idsite=${idsite}\"")
+        options.push("--data-urlencode \"url=${url}\"")
+        options.push("--data-urlencode \"event_type=${event_type}\"")
+        for(def key : ['custom3', 'custom4', 'custom5', 'custom11', 'custom12', 'custom13', 'custom14', 'custom15']){
+            if (swaCustom[key] != null) options.push("--data-urlencode \"${key}=${swaCustom[key]}\"")
+        }
+        options.push("--connect-timeout 5")
+        options.push("--max-time 20")
+
+        sh(returnStatus: true, script: "#!/bin/sh +x\ncurl ${options.join(' ')} > /dev/null 2>&1 || echo '[${parameters.get('step')}] Telemetry Report to SWA failed!'")
+
+    } catch (MissingContextVariableException noNode) {
+        echo "[${parameters.get('step')}] Telemetry Report to SWA skipped, no node available!"
+    } catch (ignore) {
+        // some error occured in SWA reporting. This should not break anything though.
+    }
+}
+

vars/artifactSetVersion.groovy

@@ -7,6 +7,7 @@ import groovy.transform.Field
 import groovy.text.SimpleTemplateEngine
 
 @Field String STEP_NAME = 'artifactSetVersion'
+@Field Set GENERAL_CONFIG_KEYS = ['collectTelemetryData']
 @Field Set STEP_CONFIG_KEYS = [
     'artifactType',
     'buildTool',
@@ -28,10 +29,7 @@ def call(Map parameters = [:]) {
 
     handlePipelineStepErrors (stepName: STEP_NAME, stepParameters: parameters) {
 
-        def gitUtils = parameters.juStabGitUtils
-        if (gitUtils == null) {
-            gitUtils = new GitUtils()
-        }
+        def gitUtils = parameters.juStabGitUtils ?: new GitUtils()
 
         if (fileExists('.git')) {
             if (sh(returnStatus: true, script: 'git diff --quiet HEAD') != 0)
@@ -43,23 +41,25 @@ def call(Map parameters = [:]) {
             script = this
 
         // load default & individual configuration
-        Map configuration = ConfigurationHelper
+        Map config = ConfigurationHelper
             .loadStepDefaults(this)
+            .mixinGeneralConfig(script.commonPipelineEnvironment, GENERAL_CONFIG_KEYS)
             .mixinStepConfig(script.commonPipelineEnvironment, STEP_CONFIG_KEYS)
+            .mixinStageConfig(script.commonPipelineEnvironment, parameters.stageName?:env.STAGE_NAME, STEP_CONFIG_KEYS)
             .mixin(gitCommitId: gitUtils.getGitCommitIdOrNull())
             .mixin(parameters, PARAMETER_KEYS)
+            .withMandatoryProperty('buildTool')
             .use()
 
-        def utils = new Utils()
-        def buildTool = utils.getMandatoryParameter(configuration, 'buildTool')
+        new Utils().pushToSWA([step: STEP_NAME, stepParam1: config.buildTool], config)
 
-        if (!configuration.filePath)
-            configuration.filePath = configuration[buildTool].filePath //use default configuration
+        if (!config.filePath)
+            config.filePath = config[config.buildTool].filePath //use default configuration
 
         def newVersion
-        def artifactVersioning = ArtifactVersioning.getArtifactVersioning(buildTool, script, configuration)
+        def artifactVersioning = ArtifactVersioning.getArtifactVersioning(config.buildTool, script, config)
 
-        if(configuration.artifactType == 'appContainer' && configuration.dockerVersionSource == 'appVersion'){
+        if(config.artifactType == 'appContainer' && config.dockerVersionSource == 'appVersion'){
             if (script.commonPipelineEnvironment.getArtifactVersion())
                 //replace + sign if available since + is not allowed in a Docker tag
                 newVersion = script.commonPipelineEnvironment.getArtifactVersion().replace('+', '_')
@@ -68,11 +68,11 @@ def call(Map parameters = [:]) {
         } else {
             def currentVersion = artifactVersioning.getVersion()
 
-            def timestamp = configuration.timestamp ? configuration.timestamp : getTimestamp(configuration.timestampTemplate)
+            def timestamp = config.timestamp ? config.timestamp : getTimestamp(config.timestampTemplate)
 
-            def versioningTemplate = configuration.versioningTemplate ? configuration.versioningTemplate : configuration[configuration.buildTool].versioningTemplate
+            def versioningTemplate = config.versioningTemplate ? config.versioningTemplate : config[config.buildTool].versioningTemplate
             //defined in default configuration
-            def binding = [version: currentVersion, timestamp: timestamp, commitId: configuration.gitCommitId]
+            def binding = [version: currentVersion, timestamp: timestamp, commitId: config.gitCommitId]
             def templatingEngine = new SimpleTemplateEngine()
             def template = templatingEngine.createTemplate(versioningTemplate).make(binding)
             newVersion = template.toString()
@@ -82,28 +82,28 @@ def call(Map parameters = [:]) {
 
         def gitCommitId
 
-        if (configuration.commitVersion) {
+        if (config.commitVersion) {
             sh 'git add .'
 
-            sshagent([configuration.gitCredentialsId]) {
+            sshagent([config.gitCredentialsId]) {
                 def gitUserMailConfig = ''
-                if (configuration.gitUserName && configuration.gitUserEMail)
-                    gitUserMailConfig = "-c user.email=\"${configuration.gitUserEMail}\" -c user.name=\"${configuration.gitUserName}\""
+                if (config.gitUserName && config.gitUserEMail)
+                    gitUserMailConfig = "-c user.email=\"${config.gitUserEMail}\" -c user.name=\"${config.gitUserName}\""
 
                 try {
                     sh "git ${gitUserMailConfig} commit -m 'update version ${newVersion}'"
                 } catch (e) {
                     error "[${STEP_NAME}]git commit failed: ${e}"
                 }
-                sh "git remote set-url origin ${configuration.gitSshUrl}"
-                sh "git tag ${configuration.tagPrefix}${newVersion}"
-                sh "git push origin ${configuration.tagPrefix}${newVersion}"
+                sh "git remote set-url origin ${config.gitSshUrl}"
+                sh "git tag ${config.tagPrefix}${newVersion}"
+                sh "git push origin ${config.tagPrefix}${newVersion}"
 
                 gitCommitId = gitUtils.getGitCommitIdOrNull()
             }
         }
 
-        if (buildTool == 'docker' && configuration.artifactType == 'appContainer') {
+        if (config.buildTool == 'docker' && config.artifactType == 'appContainer') {
             script.commonPipelineEnvironment.setAppContainerProperty('artifactVersion', newVersion)
             script.commonPipelineEnvironment.setAppContainerProperty('gitCommitId', gitCommitId)
         } else {

vars/setupCommonPipelineEnvironment.groovy

@@ -1,3 +1,10 @@
+import com.sap.piper.ConfigurationHelper
+import com.sap.piper.Utils
+import groovy.transform.Field
+
+@Field String STEP_NAME = 'setupPipelineEnvironment'
+@Field Set GENERAL_CONFIG_KEYS = ['collectTelemetryData']
+
 def call(Map parameters = [:]) {
 
     handlePipelineStepErrors (stepName: 'setupCommonPipelineEnvironment', stepParameters: parameters) {
@@ -9,6 +16,13 @@ def call(Map parameters = [:]) {
         String configFile = parameters.get('configFile')
 
         loadConfigurationFromFile(script, configFile)
+
+        Map config = ConfigurationHelper
+            .loadStepDefaults(this)
+            .mixinGeneralConfig(script.commonPipelineEnvironment, GENERAL_CONFIG_KEYS)
+            .use()
+
+        new Utils().pushToSWA([step: STEP_NAME], config)
     }
 }