diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 158edcfe6..5cfc9c0ff 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -9,6 +9,7 @@ 1. [Testing](#testing) 1. [Debugging](#debugging) 1. [Release](#release) +1. [Pipeline Configuration](#pipeline-configuration) ## Getting started @@ -428,3 +429,34 @@ We release on schedule (once a week) and on demand. To perform a release, the respective action must be invoked for which a convenience script is available in `contrib/perform-release.sh`. It requires a personal access token for GitHub with `repo` scope. Example usage `PIPER_RELEASE_TOKEN=THIS_IS_MY_TOKEN contrib/perform-release.sh`. + +## Pipeline Configuration + +The pipeline configuration is organized in a hierarchical manner and configuration parameters are incorporated from multiple sources. +In general, there are four sources for configurations: + +1. Directly passed step parameters +1. Project specific configuration placed in `.pipeline/config.yml` +1. Custom default configuration provided in `customDefaults` parameter of the project config or passed as parameter to the step `setupCommonPipelineEnvironment` +1. Default configuration from Piper library + +For more information and examples on how to configure a project, please refer to the [configuration documentation](https://sap.github.io/jenkins-library/configuration/). + +### Groovy vs. Go step configuration + +The configuration of a project is, as of now, resolved separately for Groovy and Go steps. +There are, however, dependencies between the steps responsible for resolving the configuration. +The following provides an overview of the central components and their dependencies. + +#### setupCommonPipelineEnvironment (Groovy) + +The step `setupCommonPipelineEnvironment` initializes the `commonPipelineEnvironment` and `DefaultValueCache`. +Custom default configurations can be provided as parameters to `setupCommonPipelineEnvironment` or via the `customDefaults` parameter in project configuration. + +#### DefaultValueCache (Groovy) + +The `DefaultValueCache` caches the resolved (custom) default pipeline configuration and the list of configurations that contributed to the result. +On initialization, it merges the provided custom default configurations with the default configuration from Piper library, as per the hierarchical order. + +Note, the list of configurations cached by `DefaultValueCache` is used to pass path to the (custom) default configurations of each Go step. +It only contains the paths of configurations which are **not** provided via `customDefaults` parameter of the project configuration, since the Go layer already resolves configurations provided via `customDefaults` parameter independently. diff --git a/documentation/docs/configuration.md b/documentation/docs/configuration.md index 403cb59b5..4cc9b1b1c 100644 --- a/documentation/docs/configuration.md +++ b/documentation/docs/configuration.md @@ -14,6 +14,7 @@ Configuration of the Piper steps as well the Piper templates can be done in a hi 1. Stage configuration parameters define a Jenkins pipeline stage dependent set of parameters (e.g. deployment options for the `Acceptance` stage) 1. Step configuration defines how steps behave in general (e.g. step `cloudFoundryDeploy`) 1. General configuration parameters define parameters which are available across step boundaries +1. Custom default configuration provided by the user through a reference in the `customDefaults` parameter of the project configuration 1. Default configuration comes with the Piper library and is always available ![Piper Configuration](images/piper_config.png) @@ -78,3 +79,32 @@ commonPipelineEnvironment.configuration.general.gitSshKeyCredentialsId Within library steps the `ConfigurationHelper` object is used. You can see its usage in all the Piper steps, for example [newmanExecute](https://github.com/SAP/jenkins-library/blob/master/vars/newmanExecute.groovy#L23). + +## Custom default configuration + +For projects that are composed of multiple repositories (microservices), it might be desired to provide custom default configurations. +To do that, create a YAML file which is accessible from your CI/CD environment and configure it in your project configuration. +For example, the custom default configuration can be stored in a GitHub repository and accessed via the "raw" URL: + +```yaml +customDefaults: ['https://my.github.local/raw/someorg/custom-defaults/master/backend-service.yml'] +general: + ... +``` + +Note, the parameter `customDefaults` is required to be a list of strings and needs to be defined as a separate section of the project configuration. +In addition, the item order in the list implies the precedence, i.e., the last item of the customDefaults list has highest precedence. + +It is important to ensure that the HTTP response body is proper YAML, as the pipeline will attempt to parse it. + +Anonymous read access to the `custom-defaults` repository is required. + +The custom default configuration is merged with the project's `.pipeline/config.yml`. +Note, the project's config takes precedence, so you can override the custom default configuration in your project's local configuration. +This might be useful to provide a default value that needs to be changed only in some projects. +An overview of the configuration hierarchy is given at the beginning of this page. + +If you have different types of projects, they might require different custom default configuration. +For example, you might not require all projects to have a certain code check (like Whitesource, etc.) active. +This can be achieved by having multiple YAML files in the _custom-defaults_ repository. +Configure the URL to the respective configuration file in the projects as described above. diff --git a/documentation/docs/pipelines/cloud-sdk/shared-config-between-projects.md b/documentation/docs/pipelines/cloud-sdk/shared-config-between-projects.md deleted file mode 100644 index b1c7ffe42..000000000 --- a/documentation/docs/pipelines/cloud-sdk/shared-config-between-projects.md +++ /dev/null @@ -1,26 +0,0 @@ -# Share Configuration Between Projects - -SAP Cloud SDK Pipeline does not require any programming on the application developer's end, as the pipeline is centrally developed and maintained. -The necessary configuration happens in the `.pipeline/config.yml` file in the root directory of the application's repository. - -For projects that are composed of multiple repositories (microservices), it might be desired to share the common configuration. -To do that, create a YAML file which is accessible from your CI/CD environment and configure it in your project. -For example, the common configuration can be stored in a GitHub repository an accessed via the "raw" URL: - -```yaml -general: - sharedConfiguration: 'https://my.github.local/raw/someorg/shared-config/master/backend-service.yml' -``` - -It is important to ensure that the HTTP response body is proper YAML, as the pipeline will attempt to parse it. - -Anonymous read access to the `shared-config` repository is required. - -The shared config is merged with the project's `.pipeline/config.yml`. -Note that the project's config takes precedence, so you can override the shared configuration in your project's local configuration. -This might be useful to provide a default value that needs to be changed only in some projects. - -If you have different types of projects, they might require different shared configuration. -For example, you might not require all projects to have a certain code check (like Checkmarx, SourceClear, Whitesource) active. -This can be achieved by having multiple YAML files in the _shared-config_ repository. -Configure the URL to the respective configuration file in the projects as described above. diff --git a/documentation/mkdocs.yml b/documentation/mkdocs.yml index 30375bd60..78981f0cf 100644 --- a/documentation/mkdocs.yml +++ b/documentation/mkdocs.yml @@ -24,7 +24,6 @@ nav: - 'Introduction': pipelines/cloud-sdk/introduction.md - 'Build Tools': pipelines/cloud-sdk/build-tools.md - 'Cloud Qualities': pipelines/cloud-sdk/cloud-qualities.md - - 'Shared Configuration': pipelines/cloud-sdk/shared-config-between-projects.md - 'Scenarios': - 'Build and Deploy Hybrid Applications with Jenkins and SAP Solution Manager': scenarios/changeManagement.md - 'Build and Deploy SAP UI5 or SAP Fiori Applications on SAP Cloud Platform with Jenkins': scenarios/ui5-sap-cp/Readme.md diff --git a/src/com/sap/piper/DefaultValueCache.groovy b/src/com/sap/piper/DefaultValueCache.groovy index 23c18d51f..f266707b1 100644 --- a/src/com/sap/piper/DefaultValueCache.groovy +++ b/src/com/sap/piper/DefaultValueCache.groovy @@ -1,7 +1,5 @@ package com.sap.piper -import com.sap.piper.MapUtils - @API class DefaultValueCache implements Serializable { private static DefaultValueCache instance @@ -40,24 +38,66 @@ class DefaultValueCache implements Serializable { } static void prepare(Script steps, Map parameters = [:]) { - if(parameters == null) parameters = [:] - if(!DefaultValueCache.getInstance() || parameters.customDefaults) { - def defaultValues = [:] - def configFileList = ['default_pipeline_environment.yml'] - def customDefaults = parameters.customDefaults + if (parameters == null) parameters = [:] + if (!getInstance() || parameters.customDefaults) { + List defaultsFromResources = ['default_pipeline_environment.yml'] + List customDefaults = Utils.appendParameterToStringList( + [], parameters, 'customDefaults') + defaultsFromResources.addAll(customDefaults) + List defaultsFromFiles = Utils.appendParameterToStringList( + [], parameters, 'customDefaultsFromFiles') + List defaultsFromConfig = Utils.appendParameterToStringList( + [], parameters, 'customDefaultsFromConfig') - if(customDefaults in String) - customDefaults = [customDefaults] - if(customDefaults in List) - configFileList += customDefaults - for (def configFileName : configFileList){ - if(configFileList.size() > 1) steps.echo "Loading configuration file '${configFileName}'" - def configuration = steps.readYaml text: steps.libraryResource(configFileName) - defaultValues = MapUtils.merge( - MapUtils.pruneNulls(defaultValues), - MapUtils.pruneNulls(configuration)) - } - DefaultValueCache.createInstance(defaultValues, customDefaults) + Map defaultValues = [:] + defaultValues = addDefaultsFromLibraryResources(steps, defaultValues, defaultsFromResources) + defaultValues = addDefaultsFromFiles(steps, defaultValues, defaultsFromFiles) + defaultValues = addDefaultsFromFiles(steps, defaultValues, defaultsFromConfig) + + // The "customDefault" parameter is used for storing which extra defaults need to be + // passed to piper-go. The library resource 'default_pipeline_environment.yml' shall + // be excluded, since the go steps have their own in-built defaults in their yaml files. + // And 'customDefaultsFromConfig' shall also be excluded, since piper-go handles this + // config parameter itself. + createInstance(defaultValues, customDefaults + defaultsFromFiles) } } + + private static Map addDefaultsFromLibraryResources(Script steps, Map defaultValues, List resourceFiles) { + for (String configFileName : resourceFiles) { + if (resourceFiles.size() > 1) { + steps.echo "Loading configuration file '${configFileName}'" + } + Map configuration = steps.readYaml text: steps.libraryResource(configFileName) + defaultValues = mergeIntoDefaults(defaultValues, configuration) + } + return defaultValues + } + + private static Map addDefaultsFromFiles(Script steps, Map defaultValues, List configFiles) { + for (String configFileName : configFiles) { + steps.echo "Loading configuration file '${configFileName}'" + try { + Map configuration = steps.readYaml file: ".pipeline/$configFileName" + defaultValues = mergeIntoDefaults(defaultValues, configuration) + } catch (Exception e) { + steps.error "Failed to parse custom defaults as YAML file. " + + "Please make sure it is valid YAML, and if loading from a remote location, " + + "that the response body only contains valid YAML. " + + "If you use a file from a GitHub repository, make sure you've used the 'raw' link, " + + "for example https://my.github.local/raw/someorg/shared-config/master/backend-service.yml\n" + + "File path: ${configFileName}\n" + + "Content: ${steps.readFile file: configFileName}\n" + + "Exeption message: ${e.getMessage()}\n" + + "Exception stacktrace: ${Arrays.toString(e.getStackTrace())}" + } + } + return defaultValues + } + + private static Map mergeIntoDefaults(Map defaultValues, Map configuration) { + return MapUtils.merge( + MapUtils.pruneNulls(defaultValues), + MapUtils.pruneNulls(configuration)) + } } diff --git a/src/com/sap/piper/Utils.groovy b/src/com/sap/piper/Utils.groovy index da6d2a61b..7e23ef43a 100644 --- a/src/com/sap/piper/Utils.groovy +++ b/src/com/sap/piper/Utils.groovy @@ -173,3 +173,15 @@ static String evaluateFromMavenPom(Script script, String pomFileName, String pom } return resolvedExpression } + +static List appendParameterToStringList(List list, Map parameters, String paramName) { + def value = parameters[paramName] + List result = [] + result.addAll(list) + if (value in CharSequence) { + result.add(value) + } else if (value in List) { + result.addAll(value) + } + return result +} diff --git a/test/groovy/PrepareDefaultValuesTest.groovy b/test/groovy/PrepareDefaultValuesTest.groovy index 4136ae6e1..aa0c6a7f9 100644 --- a/test/groovy/PrepareDefaultValuesTest.groovy +++ b/test/groovy/PrepareDefaultValuesTest.groovy @@ -8,9 +8,7 @@ import com.sap.piper.DefaultValueCache import util.BasePiperTest import util.JenkinsLoggingRule import util.JenkinsReadYamlRule -import util.JenkinsShellCallRule import util.JenkinsStepRule - import util.Rules public class PrepareDefaultValuesTest extends BasePiperTest { @@ -31,7 +29,7 @@ public class PrepareDefaultValuesTest extends BasePiperTest { public void setup() { helper.registerAllowedMethod("libraryResource", [String], { fileName -> - switch(fileName) { + switch (fileName) { case 'default_pipeline_environment.yml': return "default: 'config'" case 'custom.yml': return "custom: 'myConfig'" case 'not_found': throw new hudson.AbortException('No such library resource not_found could be found') diff --git a/test/groovy/SetupCommonPipelineEnvironmentTest.groovy b/test/groovy/SetupCommonPipelineEnvironmentTest.groovy index ddf439876..09bdb2d73 100644 --- a/test/groovy/SetupCommonPipelineEnvironmentTest.groovy +++ b/test/groovy/SetupCommonPipelineEnvironmentTest.groovy @@ -1,15 +1,22 @@ +import com.sap.piper.DefaultValueCache import org.junit.Before import org.junit.Rule import org.junit.Test +import org.junit.rules.ExpectedException import org.junit.rules.RuleChain import org.yaml.snakeyaml.Yaml import util.BasePiperTest +import util.JenkinsReadFileRule +import util.JenkinsShellCallRule import util.JenkinsStepRule import util.JenkinsWriteFileRule import util.Rules +import static org.hamcrest.Matchers.hasItem import static org.junit.Assert.assertEquals import static org.junit.Assert.assertNotNull +import static org.junit.Assert.assertThat +import static org.junit.Assert.assertTrue class SetupCommonPipelineEnvironmentTest extends BasePiperTest { @@ -17,22 +24,43 @@ class SetupCommonPipelineEnvironmentTest extends BasePiperTest { private JenkinsStepRule stepRule = new JenkinsStepRule(this) private JenkinsWriteFileRule writeFileRule = new JenkinsWriteFileRule(this) + private ExpectedException thrown = ExpectedException.none() + private JenkinsShellCallRule shellRule = new JenkinsShellCallRule(this) + private JenkinsReadFileRule readFileRule = new JenkinsReadFileRule(this, "./") @Rule public RuleChain rules = Rules .getCommonRules(this) .around(stepRule) .around(writeFileRule) + .around(thrown) + .around(shellRule) + .around(readFileRule) + @Before void init() { def examplePipelineConfig = new File('test/resources/test_pipeline_config.yml').text + helper.registerAllowedMethod("libraryResource", [String], { fileName -> + switch(fileName) { + case 'default_pipeline_environment.yml': return "default: 'config'" + case 'custom.yml': return "custom: 'myConfig'" + case 'notFound.yml': throw new hudson.AbortException('No such library resource notFound could be found') + default: return "the:'end'" + } + }) + helper.registerAllowedMethod("readYaml", [Map], { Map parameters -> Yaml yamlParser = new Yaml() if (parameters.text) { return yamlParser.load(parameters.text) + } else if(parameters.file) { + if(parameters.file == '.pipeline/default_pipeline_environment.yml') return [default: 'config'] + else if (parameters.file == '.pipeline/custom.yml') return [custom: 'myConfig'] + } else { + throw new IllegalArgumentException("Key 'text' and 'file' are both missing in map ${m}.") } usedConfigFile = parameters.file return yamlParser.load(examplePipelineConfig) @@ -68,5 +96,77 @@ class SetupCommonPipelineEnvironmentTest extends BasePiperTest { assertEquals('develop', nullScript.commonPipelineEnvironment.configuration.general.productiveBranch) assertEquals('my-maven-docker', nullScript.commonPipelineEnvironment.configuration.steps.mavenExecute.dockerImage) } + + @Test + public void testAttemptToLoadNonExistingConfigFile() { + + helper.registerAllowedMethod("fileExists", [String], { String path -> + switch(path) { + case 'default_pipeline_environment.yml': return false + case 'custom.yml': return false + case 'notFound.yml': return false + default: return true + } + }) + + helper.registerAllowedMethod("handlePipelineStepErrors", [Map,Closure], { Map map, Closure closure -> + closure() + }) + + // Behavior documented here based on reality check + thrown.expect(hudson.AbortException.class) + thrown.expectMessage('No such library resource notFound could be found') + + stepRule.step.setupCommonPipelineEnvironment( + script: nullScript, + customDefaults: 'notFound.yml' + ) + } + + @Test + void testAttemptToLoadFileFromURL() { + helper.registerAllowedMethod("fileExists", [String], {String path -> + switch (path) { + case 'default_pipeline_environment.yml': return false + default: return true + } + }) + + String customDefaultUrl = "https://url-to-my-config.com/my-config.yml" + boolean urlRequested = false + + helper.registerAllowedMethod("httpRequest", [Map], {Map parameters -> + switch (parameters.url) { + case customDefaultUrl: + urlRequested = true + return [status: 200, content: "custom: 'myRemoteConfig'"] + default: + throw new IllegalArgumentException('wrong URL requested') + } + }) + + helper.registerAllowedMethod("readYaml", [Map], { Map parameters -> + Yaml yamlParser = new Yaml() + if (parameters.text) { + return yamlParser.load(parameters.text) + } else if (parameters.file) { + if (parameters.file == '.pipeline/config-with-custom-defaults.yml') { + return [customDefaults: "${customDefaultUrl}"] + } + if (parameters.file == '.pipeline/custom_default_from_url_0.yml') { + return [custom: 'myRemoteConfig'] + } + } + throw new IllegalArgumentException("Unexpected invocation of readYaml step") + }) + + stepRule.step.setupCommonPipelineEnvironment( + script: nullScript, + customDefaults: 'custom.yml', + configFile: '.pipeline/config-with-custom-defaults.yml', + ) + assertEquals("custom: 'myRemoteConfig'", writeFileRule.files['.pipeline/custom_default_from_url_0.yml']) + assertEquals('myRemoteConfig', DefaultValueCache.instance.defaultValues['custom']) + } } diff --git a/vars/prepareDefaultValues.groovy b/vars/prepareDefaultValues.groovy index 7dfa6ec05..ec9e3fc09 100644 --- a/vars/prepareDefaultValues.groovy +++ b/vars/prepareDefaultValues.groovy @@ -1,9 +1,10 @@ import com.sap.piper.GenerateDocumentation import com.sap.piper.DefaultValueCache -import com.sap.piper.MapUtils import groovy.transform.Field +import static com.sap.piper.Prerequisites.checkScript + @Field STEP_NAME = getClass().getName() @Field Set GENERAL_CONFIG_KEYS = [] @@ -16,5 +17,6 @@ import groovy.transform.Field */ @GenerateDocumentation void call(Map parameters = [:]) { - DefaultValueCache.prepare(this, parameters) + def script = checkScript(this, parameters) + DefaultValueCache.prepare(script, parameters) } diff --git a/vars/setupCommonPipelineEnvironment.groovy b/vars/setupCommonPipelineEnvironment.groovy index 605a1c7f8..12d48c274 100644 --- a/vars/setupCommonPipelineEnvironment.groovy +++ b/vars/setupCommonPipelineEnvironment.groovy @@ -17,8 +17,17 @@ import groovy.transform.Field @Field Set STEP_CONFIG_KEYS = [] @Field Set PARAMETER_KEYS = [ - /** Property file defining project specific settings.*/ - 'configFile' + /** Path to the pipeline configuration file defining project specific settings.*/ + 'configFile', + /** A list of file names which will be extracted from library resources and which serve as source for + * default values for the pipeline configuration. These are merged with and override built-in defaults, with + * a parameter supplied by the last resource file taking precedence over the same parameter supplied in an + * earlier resource file or built-in default.*/ + 'customDefaults', + /** A list of file paths or URLs which must point to YAML content. These work exactly like + * `customDefaults`, but from local or remote files instead of library resources. They are merged with and + * take precedence over `customDefaults`.*/ + 'customDefaultsFromFiles' ] /** @@ -35,20 +44,43 @@ void call(Map parameters = [:]) { def script = checkScript(this, parameters) - prepareDefaultValues script: script, customDefaults: parameters.customDefaults + String configFile = parameters.get('configFile') + loadConfigurationFromFile(script, configFile) - List customDefaults = ['default_pipeline_environment.yml'].plus(parameters.customDefaults?:[]) - customDefaults.each { + // Copy custom defaults from library resources to include them in the 'pipelineConfigAndTests' stash + List customDefaultsResources = Utils.appendParameterToStringList( + ['default_pipeline_environment.yml'], parameters, 'customDefaults') + customDefaultsResources.each { cd -> writeFile file: ".pipeline/${cd}", text: libraryResource(cd) } + List customDefaultsFiles = Utils.appendParameterToStringList( + [], parameters, 'customDefaultsFromFiles') + customDefaultsFiles = copyOrDownloadCustomDefaultsIntoPipelineEnv(script, customDefaultsFiles) + + List customDefaultsConfig = [] + if (script.commonPipelineEnvironment.configuration.customDefaults) { + if (!script.commonPipelineEnvironment.configuration.customDefaults in List) { + // Align with Go side on supported parameter type. + error "You have defined the parameter 'customDefaults' in your project configuration " + + "but it is of an unexpected type. Please make sure that it is a list of strings, i.e. " + + "customDefaults = ['...']. See https://sap.github.io/jenkins-library/configuration/ for " + + "more details." + } + customDefaultsConfig = Utils.appendParameterToStringList( + [], script.commonPipelineEnvironment.configuration as Map, 'customDefaults') + } + customDefaultsConfig = copyOrDownloadCustomDefaultsIntoPipelineEnv(script, customDefaultsConfig) + + prepareDefaultValues([ + script: script, + customDefaults: parameters.customDefaults, + customDefaultsFromFiles: customDefaultsFiles, + customDefaultsFromConfig: customDefaultsConfig ]) + stash name: 'pipelineConfigAndTests', includes: '.pipeline/**', allowEmpty: true - String configFile = parameters.get('configFile') - - loadConfigurationFromFile(script, configFile) - Map config = ConfigurationHelper.newInstance(this) .loadStepDefaults() .mixinGeneralConfig(script.commonPipelineEnvironment, GENERAL_CONFIG_KEYS) @@ -65,20 +97,53 @@ void call(Map parameters = [:]) { } } -private loadConfigurationFromFile(script, String configFile) { +private static loadConfigurationFromFile(script, String configFile) { if (!configFile) { String defaultYmlConfigFile = '.pipeline/config.yml' String defaultYamlConfigFile = '.pipeline/config.yaml' - if (fileExists(defaultYmlConfigFile)) { + if (script.fileExists(defaultYmlConfigFile)) { configFile = defaultYmlConfigFile - } else if (fileExists(defaultYamlConfigFile)) { + } else if (script.fileExists(defaultYamlConfigFile)) { configFile = defaultYamlConfigFile } } // A file passed to the function is not checked for existence in order to fail the pipeline. if (configFile) { - script.commonPipelineEnvironment.configuration = readYaml(file: configFile) + script.commonPipelineEnvironment.configuration = script.readYaml(file: configFile) script.commonPipelineEnvironment.configurationFile = configFile } } + +private static List copyOrDownloadCustomDefaultsIntoPipelineEnv(script, List customDefaults) { + List fileList = [] + int urlCount = 0 + for (int i = 0; i < customDefaults.size(); i++) { + // copy retrieved file to .pipeline/ to make sure they are in the pipelineConfigAndTests stash + String fileName + if (customDefaults[i].startsWith('http://') || customDefaults[i].startsWith('https://')) { + fileName = "custom_default_from_url_${urlCount}.yml" + + def response = script.httpRequest( + url: customDefaults[i], + validResponseCodes: '100:399,404' // Allow a more specific error message for 404 case + ) + if (response.status == 404) { + error "URL for remote custom defaults (${customDefaults[i]}) appears to be incorrect. " + + "Server returned HTTP status code 404. " + + "Please make sure that the path is correct and no authentication is required to retrieve the file." + } + + script.writeFile file: ".pipeline/$fileName", text: response.content + urlCount++ + } else if (script.fileExists(customDefaults[i])) { + fileName = customDefaults[i] + script.writeFile file: ".pipeline/$fileName", text: script.readFile(file: fileName) + } else { + script.echo "WARNING: Custom default entry not found: '${customDefaults[i]}', it will be ignored" + continue + } + fileList.add(fileName) + } + return fileList +}