add step gaugeExecuteTests (#340)

* seleniumExecuteTests - fixes * add step gaugeExecuteTests incl. tests * add documentation * add more config options
2024-11-28 08:49:44 +02:00 · 2018-10-17 16:44:20 +02:00 · 2018-10-17 16:44:20 +02:00 · 0e5ccabdae
commit 0e5ccabdae
parent b657dc6a29
8 changed files with 384 additions and 2 deletions
--- a/documentation/docs/steps/gaugeExecuteTests.md
+++ b/documentation/docs/steps/gaugeExecuteTests.md
@ -0,0 +1,102 @@
+# gaugeExecuteTests
+
+## Description
+In this step Gauge ([getgauge.io](http:getgauge.io)) acceptance tests are executed.
+Using Gauge it will be possible to have a three-tier test layout:
+* Acceptance Criteria
+* Test implemenation layer
+* Application driver layer
+
+This layout is propagated by Jez Humble and Dave Farley in their book "Continuous Delivery" as a way to create maintainable acceptance test suites (see "Continuous Delivery", p. 190ff).
+
+Using Gauge it is possible to write test specifications in [Markdown syntax](http://daringfireball.net/projects/markdown/syntax) and therefore allow e.g. product owners to write the relevant acceptance test specifications. At the same time it allows the developer to implement the steps described in the specification in her development environment.
+
+You can use the sample projects of Gauge, for example: https://github.com/getgauge/gauge-mvn-archetypes
+
+!!! note "Make sure to run against a Selenium Hub configuration"
+    In the test example of _gauge-archetype-selenium_ please make sure to allow it to run against a Selenium hub:
+
+    Please extend DriverFactory.java for example in following way:
+
+    ``` java
+    String hubUrl = System.getenv("HUB_URL");
+    //when running on a Docker deamon (and not using Kubernetes plugin), Docker images will be linked
+    //in this case hubUrl will be http://selenium:4444/wd/hub due to the linking of the containers
+    hubUrl = (hubUrl == null) ? "http://localhost:4444/wd/hub" : hubUrl;
+    Capabilities chromeCapabilities = DesiredCapabilities.chrome();
+    System.out.println("Running on Selenium Hub: " + hubUrl);
+    return new RemoteWebDriver(new URL(hubUrl), chromeCapabilities);
+    ```
+
+## Prerequsites
+
+none
+
+## Example
+
+Pipeline step:
+```groovy
+gaugeExecuteTests script: this, testServerUrl: 'http://test.url'
+```
+
+
+## Parameters
+
+| parameter | mandatory | default | possible values |
+| ----------|-----------|---------|-----------------|
+|script|yes|||
+|buildTool|no|`maven`||
+|dockerEnvVars|no|`[HUB:TRUE, HUB_URL:http://localhost:4444/wd/hub]`||
+|dockerImage|no|buildTool=`maven`: `maven:3.5-jdk-8`<br />buildTool=`npm`: `node:8-stretch`<br />||
+|dockerName|no|buildTool=`maven`: `maven`<br />buildTool=`npm`: `npm`<br />||
+|dockerWorkspace|no|buildTool=`maven`: ``<br />buildTool=`npm`: `/home/node`<br />||
+|failOnError|no|`false`||
+|gitBranch|no|||
+|gitSshKeyCredentialsId|no|``||
+|installCommand|no|`curl -SsL https://downloads.gauge.org/stable | sh -s -- --location=$HOME/bin/gauge`||
+|languageRunner|no|buildTool=`maven`: `java`<br />buildTool=`npm`: `js`<br />||
+|runCommand|no|buildTool=`maven`: `mvn test-compile gauge:execute`<br />buildTool=`npm`: `gauge run`<br />||
+|stashContent|no|<ul><li>`buildDescriptor`</li><li>`tests`</li></ul>||
+|testOptions|no|buildTool=`maven`: `-DspecsDir=specs`<br />buildTool=`npm`: `specs`<br />||
+|testRepository|no|||
+|testServerUrl|no|||
+
+
+Details:
+
+* `script` defines the global script environment of the Jenkinsfile run. Typically `this` is passed to this parameter. This allows the function to access the [`commonPipelineEnvironment`](commonPipelineEnvironment.md) for storing the measured duration.
+* `buildTool` defines the build tool to be used for the test execution.
+* `dockerEnvVars`, see step [dockerExecute](dockerExecute.md)
+* `dockerImage`, see step [dockerExecute](dockerExecute.md)
+* `dockerName`, see step [dockerExecute](dockerExecute.md)
+* `dockerWorkspace`, see step [dockerExecute](dockerExecute.md)
+* With `failOnError` you can define the behavior, in case tests fail. When this is set to `true` test results cannot be recorded using the `publishTestResults` step afterwards.
+* `installCommand` defines the command for installing Gauge. In case the `dockerImage` already contains Gauge it can be set to empty: ``.
+* `languageRunner` defines the Gauge language runner to be used.
+* `runCommand` defines the command which is used for executing Gauge.
+* If specific stashes should be considered for the tests, you can pass this via parameter `stashContent`
+* `testOptions` allows to set specific options for the Gauge execution. Details can be found for example [in the Gauge Maven plugin documentation](https://github.com/getgauge/gauge-maven-plugin#executing-specs)
+* In case the test implementation is stored in a different repository than the code itself, you can define the repository containing the tests using parameter `testRepository` and if required `gitBranch` (for a different branch than master) and `gitSshKeyCredentialsId` (for protected repositories). For protected repositories the `testRepository` needs to contain the ssh git url.
+* `testServerUrl` is passed as environment variable `TARGET_SERVER_URL` to the test execution. Tests running against the system should read the host information from this environment variable in order to be infrastructure agnostic.
+
+## Step configuration
+
+We recommend to define values of step parameters via [config.yml file](../configuration.md).
+
+In following sections the configuration is possible:
+
+| parameter | general | step | stage |
+| ----------|-----------|---------|-----------------|
+|script||||
+|buildTool||X|X|
+|dockerEnvVars||X|X|
+|dockerImage||X|X|
+|dockerName||X|X|
+|dockerWorkspace||X|X|
+|failOnError||X|X|
+|gitBranch||X|X|
+|gitSshKeyCredentialsId||X|X|
+|stashContent||X|X|
+|testOptions||X|X|
+|testRepository||X|X|
+|testServerUrl||X|X|
--- a/documentation/docs/steps/seleniumExecuteTests.md
+++ b/documentation/docs/steps/seleniumExecuteTests.md
@ -77,8 +77,10 @@ webdriverio
 | parameter | mandatory | default | possible values |
 | ----------|-----------|---------|-----------------|
 |script|yes|||
+|buildTool|no|`npm`|`maven`, `npm`|
 |containerPortMappings|no|`[selenium/standalone-chrome:[[containerPort:4444, hostPort:4444]]]`||
-|dockerImage|no|buildTool=`maven`: `maven:3.5-jdk-7`<br />buildTool=`npm`: `node:8-stretch`<br />||
+|dockerEnvVars|no|||
+|dockerImage|no|buildTool=`maven`: `maven:3.5-jdk-8`<br />buildTool=`npm`: `node:8-stretch`<br />||
 |dockerName|no|buildTool=`maven`: `maven`<br />buildTool=`npm`: `npm`<br />||
 |dockerWorkspace|no|buildTool=`maven`: ``<br />buildTool=`npm`: `/home/node`<br />||
 |failOnError|no|`true`||
@ -92,7 +94,9 @@ webdriverio
 |testRepository|no|||

 * `script` defines the global script environment of the Jenkinsfile run. Typically `this` is passed to this parameter. This allows the function to access the [`commonPipelineEnvironment`](commonPipelineEnvironment.md) for storing the measured duration.
+* `buildTool` defines the build tool to be used for the test execution.
 * `containerPortMappings`, see step [dockerExecute](dockerExecute.md)
+* `dockerEnvVars`, see step [dockerExecute](dockerExecute.md)
 * `dockerImage`, see step [dockerExecute](dockerExecute.md)
 * `dockerName`, see step [dockerExecute](dockerExecute.md)
 * `dockerWorkspace`, see step [dockerExecute](dockerExecute.md)
@ -113,7 +117,9 @@ In following sections the configuration is possible:
 | parameter | general | step | stage |
 | ----------|-----------|---------|-----------------|
 |script||||
+|buildTool||X|X|
 |containerPortMappings|X|X|X|
+|dockerEnvVars|X|X|X|
 |dockerImage|X|X|X|
 |dockerName|X|X|X|
 |dockerWorkspace|X|X|X|
--- a/documentation/mkdocs.yml
+++ b/documentation/mkdocs.yml
@ -12,6 +12,7 @@ nav:
        - dockerExecute: steps/dockerExecute.md
        - dockerExecuteOnKubernetes: steps/dockerExecuteOnKubernetes.md
        - durationMeasure: steps/durationMeasure.md
+        - gaugeExecuteTests: steps/gaugeExecuteTests.md
        - handlePipelineStepErrors: steps/handlePipelineStepErrors.md
        - healthExecuteCheck: steps/healthExecuteCheck.md
        - influxWriteData: steps/influxWriteData.md
--- a/resources/default_pipeline_environment.yml
+++ b/resources/default_pipeline_environment.yml
@ -139,6 +139,30 @@ steps:
      workspace: '**/*.*'
    stashExcludes:
      workspace: 'nohup.out'
+  gaugeExecuteTests:
+    buildTool: 'maven'
+    dockerEnvVars:
+      HUB: 'TRUE'
+      HUB_URL: 'http://localhost:4444/wd/hub'
+    failOnError: false
+    installCommand: 'curl -SsL https://downloads.gauge.org/stable | sh -s -- --location=$HOME/bin/gauge'
+    stashContent:
+      - 'buildDescriptor'
+      - 'tests'
+    maven:
+      dockerImage: 'maven:3.5-jdk-8'
+      dockerName: 'maven'
+      dockerWorkspace: ''
+      languageRunner: 'java'
+      runCommand: 'mvn test-compile gauge:execute'
+      testOptions: '-DspecsDir=specs'
+    npm:
+      dockerImage: 'node:8-stretch'
+      dockerName: 'npm'
+      dockerWorkspace: '/home/node'
+      languageRunner: 'js'
+      runCommand: 'gauge run'
+      testOptions: 'specs'
  healthExecuteCheck:
    healthEndpoint: ''
  influxWriteData:
@ -214,7 +238,7 @@ steps:
    stashContent:
      - 'tests'
    maven:
-      dockerImage: 'maven:3.5-jdk-7'
+      dockerImage: 'maven:3.5-jdk-8'
      dockerName: 'maven'
      dockerWorkspace: ''
    npm:
--- a/test/groovy/GaugeExecuteTestsTest.groovy
+++ b/test/groovy/GaugeExecuteTestsTest.groovy
@ -0,0 +1,137 @@
+#!groovy
+import org.junit.Before
+import org.junit.Rule
+import org.junit.Test
+import org.junit.rules.ExpectedException
+import org.junit.rules.RuleChain
+import util.*
+
+import static org.hamcrest.Matchers.*
+import static org.junit.Assert.assertThat
+
+class GaugeExecuteTestsTest extends BasePiperTest {
+    private JenkinsStepRule jsr = new JenkinsStepRule(this)
+    private JenkinsLoggingRule jlr = new JenkinsLoggingRule(this)
+    private JenkinsShellCallRule jscr = new JenkinsShellCallRule(this)
+    private JenkinsEnvironmentRule jer = new JenkinsEnvironmentRule(this)
+    private ExpectedException thrown = ExpectedException.none()
+
+    @Rule
+    public RuleChain rules = Rules
+        .getCommonRules(this)
+        .around(new JenkinsReadYamlRule(this))
+        .around(jscr)
+        .around(jlr)
+        .around(jer)
+        .around(jsr)
+        .around(thrown)
+
+    def gitParams = [:]
+    def seleniumParams = [:]
+
+    @Before
+    void init() throws Exception {
+        helper.registerAllowedMethod("git", [Map.class], { map -> gitParams = map })
+        helper.registerAllowedMethod("unstash", [String.class], { s -> return [s]})
+
+        helper.registerAllowedMethod('seleniumExecuteTests', [Map.class, Closure.class], {map, body ->
+            seleniumParams = map
+            return body()
+        })
+    }
+
+    @Test
+    void testExecuteGaugeDefaultSuccess() throws Exception {
+        jsr.step.gaugeExecuteTests(
+            script: nullScript,
+            juStabUtils: utils,
+            testServerUrl: 'http://test.url'
+        )
+        assertThat(jscr.shell, hasItem(stringContainsInOrder([
+            'export HOME=${HOME:-$(pwd)}',
+            'if [ "$HOME" = "/" ]; then export HOME=$(pwd); fi',
+            'export PATH=$HOME/bin/gauge:$PATH',
+            'mkdir -p $HOME/bin/gauge',
+            'curl -SsL https://downloads.gauge.org/stable | sh -s -- --location=$HOME/bin/gauge',
+            'gauge telemetry off',
+            'gauge install java',
+            'gauge install html-report',
+            'gauge install xml-report',
+            'mvn test-compile gauge:execute -DspecsDir=specs'
+        ])))
+        assertThat(seleniumParams.dockerImage, is('maven:3.5-jdk-8'))
+        assertThat(seleniumParams.dockerEnvVars, hasEntry('TARGET_SERVER_URL', 'http://test.url'))
+        assertThat(seleniumParams.dockerName, is('maven'))
+        assertThat(seleniumParams.dockerWorkspace, is(''))
+        assertThat(seleniumParams.stashContent, hasSize(2))
+        assertThat(seleniumParams.stashContent, allOf(hasItem('buildDescriptor'), hasItem('tests')))
+        assertJobStatusSuccess()
+    }
+
+    @Test
+    void testExecuteGaugeNode() throws Exception {
+        jsr.step.gaugeExecuteTests(
+            script: nullScript,
+            buildTool: 'npm',
+            dockerEnvVars: ['TARGET_SERVER_URL':'http://custom.url'],
+            juStabUtils: utils,
+            testOptions: 'testSpec'
+        )
+        assertThat(jscr.shell, hasItem(stringContainsInOrder([
+            'gauge install js',
+            'gauge run testSpec'
+        ])))
+        assertThat(seleniumParams.dockerImage, is('node:8-stretch'))
+        assertThat(seleniumParams.dockerEnvVars, hasEntry('TARGET_SERVER_URL', 'http://custom.url'))
+        assertThat(seleniumParams.dockerName, is('npm'))
+        assertThat(seleniumParams.dockerWorkspace, is('/home/node'))
+        assertJobStatusSuccess()
+    }
+
+    @Test
+    void testExecuteCustomWithError() throws Exception {
+        helper.registerAllowedMethod("sh", [String.class], { s ->
+            throw new RuntimeException('Test Error')
+        })
+        thrown.expect(RuntimeException)
+        thrown.expectMessage('Test Error')
+        try {
+            jsr.step.gaugeExecuteTests(
+                script: nullScript,
+                juStabUtils: utils,
+                dockerImage: 'testImage',
+                dockerName: 'testImageName',
+                dockerWorkspace: '/home/test',
+                failOnError: true,
+                stashContent: ['testStash'],
+            )
+
+        } finally{
+            assertThat(seleniumParams.dockerImage, is('testImage'))
+            assertThat(seleniumParams.dockerName, is('testImageName'))
+            assertThat(seleniumParams.dockerWorkspace, is('/home/test'))
+            assertThat(seleniumParams.stashContent, hasSize(1))
+            assertThat(jlr.log, containsString('[gaugeExecuteTests] One or more tests failed'))
+            assertThat(nullScript.currentBuild.result, is('UNSTABLE'))
+
+        }
+
+    }
+
+    @Test
+    void testExecuteGaugeCustomRepo() throws Exception {
+        helper.registerAllowedMethod('git', [String.class], null)
+        helper.registerAllowedMethod('stash', [String.class], null)
+
+        jsr.step.gaugeExecuteTests(
+            script: nullScript,
+            juStabUtils: utils,
+            testRepository: 'myTestRepo',
+            failOnError: true
+        )
+
+        // nested matchers do not work correctly
+        assertThat(seleniumParams.stashContent, hasItem(startsWith('testContent-')))
+        assertJobStatusSuccess()
+    }
+}
--- a/test/groovy/SeleniumExecuteTestsTest.groovy
+++ b/test/groovy/SeleniumExecuteTestsTest.groovy
@ -47,6 +47,7 @@ class SeleniumExecuteTestsTest extends BasePiperTest {
        }
        assertThat(bodyExecuted, is(true))
        assertThat(jedr.dockerParams.containerPortMappings, is(['selenium/standalone-chrome': [[containerPort: 4444, hostPort: 4444]]]))
+        assertThat(jedr.dockerParams.dockerEnvVars, is(null))
        assertThat(jedr.dockerParams.dockerImage, is('node:8-stretch'))
        assertThat(jedr.dockerParams.dockerName, is('npm'))
        assertThat(jedr.dockerParams.dockerWorkspace, is('/home/node'))
@ -56,6 +57,20 @@ class SeleniumExecuteTestsTest extends BasePiperTest {
        assertThat(jedr.dockerParams.sidecarVolumeBind, is(['/dev/shm': '/dev/shm']))
    }

+    @Test
+    void testExecuteSeleniumCustomBuildTool() {
+        jsr.step.seleniumExecuteTests(
+            script: nullScript,
+            buildTool: 'maven',
+            juStabUtils: utils
+        ) {
+            bodyExecuted = true
+        }
+        assertThat(bodyExecuted, is(true))
+        assertThat(jedr.dockerParams.dockerImage, is('maven:3.5-jdk-8'))
+        assertThat(jedr.dockerParams.dockerName, is('maven'))
+        assertThat(jedr.dockerParams.dockerWorkspace, is(''))
+    }
    @Test
    void testExecuteSeleniumError() {
        thrown.expectMessage('Error occured')
--- a/vars/gaugeExecuteTests.groovy
+++ b/vars/gaugeExecuteTests.groovy
@ -0,0 +1,94 @@
+import com.sap.piper.Utils
+import com.sap.piper.ConfigurationHelper
+import com.sap.piper.GitUtils
+import groovy.text.SimpleTemplateEngine
+import groovy.transform.Field
+
+@Field String STEP_NAME = 'gaugeExecuteTests'
+@Field Set STEP_CONFIG_KEYS = [
+    'buildTool',
+    'dockerEnvVars',
+    'dockerImage',
+    'dockerName',
+    'dockerWorkspace',
+    'failOnError',
+    'gitBranch',
+    'gitSshKeyCredentialsId',
+    'installCommand',
+    'languageRunner',
+    'runCommand',
+    'stashContent',
+    'testOptions',
+    'testRepository',
+    'testServerUrl'
+]
+@Field Set PARAMETER_KEYS = STEP_CONFIG_KEYS
+
+void call(Map parameters = [:]) {
+    handlePipelineStepErrors (stepName: STEP_NAME, stepParameters: parameters) {
+        def script = parameters.script ?: [commonPipelineEnvironment: commonPipelineEnvironment]
+        def utils = parameters.juStabUtils ?: new Utils()
+
+        script.commonPipelineEnvironment.setInfluxStepData('gauge', false)
+
+        // load default & individual configuration
+        Map config = ConfigurationHelper
+            .loadStepDefaults(this)
+            .mixinStepConfig(script.commonPipelineEnvironment, STEP_CONFIG_KEYS)
+            .mixinStageConfig(script.commonPipelineEnvironment, parameters.stageName?:env.STAGE_NAME, STEP_CONFIG_KEYS)
+            .mixin(parameters, PARAMETER_KEYS)
+            .dependingOn('buildTool').mixin('dockerImage')
+            .dependingOn('buildTool').mixin('dockerName')
+            .dependingOn('buildTool').mixin('dockerWorkspace')
+            .dependingOn('buildTool').mixin('languageRunner')
+            .dependingOn('buildTool').mixin('runCommand')
+            .dependingOn('buildTool').mixin('testOptions')
+            .use()
+
+        utils.pushToSWA([step: STEP_NAME, stepParam1: config.buildTool, stepParam2: config.dockerName], config)
+
+        if(!config.dockerEnvVars.TARGET_SERVER_URL && config.testServerUrl)
+            config.dockerEnvVars.TARGET_SERVER_URL = config.testServerUrl
+
+        if (config.testRepository) {
+            // handle separate test repository
+            config.stashContent = [GitUtils.handleTestRepository(this, config)]
+        } else {
+            config.stashContent = utils.unstashAll(config.stashContent)
+        }
+
+        seleniumExecuteTests (
+            script: script,
+            buildTool: config.buildTool,
+            dockerEnvVars: config.dockerEnvVars,
+            dockerImage: config.dockerImage,
+            dockerName: config.dockerName,
+            dockerWorkspace: config.dockerWorkspace,
+            stashContent: config.stashContent
+        ) {
+            String gaugeScript = ''
+            if (config.installCommand) {
+                gaugeScript = '''export HOME=${HOME:-$(pwd)}
+                    if [ "$HOME" = "/" ]; then export HOME=$(pwd); fi
+                    export PATH=$HOME/bin/gauge:$PATH
+                    mkdir -p $HOME/bin/gauge
+                    ''' + config.installCommand + '''
+                    gauge telemetry off
+                    gauge install ''' + config.languageRunner + '''
+                    gauge install html-report
+                    gauge install xml-report
+                    '''
+            }
+            gaugeScript += config.runCommand
+
+            try {
+                sh "${gaugeScript} ${config.testOptions}"
+                script.commonPipelineEnvironment.setInfluxStepData('gauge', true)
+            } catch (err) {
+                echo "[${STEP_NAME}] One or more tests failed"
+                script.currentBuild.result = 'UNSTABLE'
+                if (config.failOnError) throw err
+            }
+        }
+    }
+}
--- a/vars/seleniumExecuteTests.groovy
+++ b/vars/seleniumExecuteTests.groovy
@ -7,7 +7,9 @@ import groovy.text.SimpleTemplateEngine

@Field String STEP_NAME = 'seleniumExecuteTests'
@Field Set STEP_CONFIG_KEYS = [
+    'buildTool', //defines the tool which is used for executing the tests
    'containerPortMappings', //port mappings required for containers. This will only take effect inside a Kubernetes pod, format [[containerPort: 1111, hostPort: 1111]]
+    'dockerEnvVars', //envVars to be set in the execution container if required
    'dockerImage', //Docker image for code execution
    'dockerName', //name of the Docker container. This will only take effect inside a Kubernetes pod.
    'dockerWorkspace', //user home directory for Docker execution. This will only take effect inside a Kubernetes pod.
@ -45,6 +47,7 @@ def call(Map parameters = [:], Closure body) {
        dockerExecute(
                script: script,
                containerPortMappings: config.containerPortMappings,
+                dockerEnvVars: config.dockerEnvVars,
                dockerImage: config.dockerImage,
                dockerName: config.dockerName,
                dockerWorkspace: config.dockerWorkspace,