jenkins/sap-jenkins-library
1
0
Fork 0
mirror of https://github.com/SAP/jenkins-library.git synced 2025-10-30 23:57:50 +02:00
Code Issues Releases Activity

Variable Substitution in YAML Files (#852)

Browse Source 
* Changes:
- New YamlSubstituteVariables step to substitute variables in YAML files with values from another YAML
- New Tests, that check the different substitution patterns.
- Added test resources, including various manifest and variables files.
- Improved usage of JenkinsLoggingRule
- Improved JenkinsReadYamlRule to properly reflect the mocked library's behaviour.
- Added a new JenkinsWriteYamlRule.

* Changes:
- added a Logger that checks a config.verbose flag before it logs debug messages.
- changed error handling to rethrow Yaml parsing exception in case of wrongly-formatted Yaml files.
- changed JenkinsWriteYamlRule to capture Yaml file details of every invocation of writeYaml. This allows sanity checks at end of tests, even if there were multiple invocations.
- adjusted tests.

* Changes:
- Removed javadoc-code blocks from API documentation since they are not supported.
- Removed skipDeletion boolean.
- Added a new deleteFile script which deletes a file if present.
- Added a new JenkinsDeleteFileRule to mock deleteFile script and optionally skip deletion for tests.
- Adjusted yamlSubstituteVariables script.
- Adjusted tests to include new JenkinsDeleteFileRule.
- Changed code that deletes an already existing output file to produce better logs.

* Changes:
- Turned yamlSubstituteVariables into a script that works purely based on Yaml data (not files).
- Added a new cfManifestSubstituteVariables that uses yamlSubstituteVariables under the hood but works based on files.
- Adjusted tests, and added new ones.

* Adjusted documentation and a few log statements.

* Changed documentation to no longer include javadoc code statements.

* Made mocking of deletion of a file a default. Adjusted tests.

* Changed signature of yamlSubstituteVariables' call method to return void.

* Changes:
- Fixed naming issues in deleteFile.
- Renamed Logger to DebugHelper.
- Fixed some documentation.

* Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.

* PROPERLY Changed implementation of deleteFile not to use java.io.File - which is evil when using it for file operations.

* Changes:
- Added tests for deleteFile script
- Changed JenkinsFileExistsRule to also keep track of which files have been queried for existence.

* Changes:
- Removed java.io.File usage from cfManifestSubstituteVariables and using fileExists instead now.
- Adjusted tests.

* Wrapped file path inside ticks to allow spaces in file path when calling deleteFile.

* Removed null checks of mandatory parameters, and resorted to ConfigurationHelper.withMandatoryProperty

* Fixed a NullPointer due to weird Jenkins / Groovy behaviour.

* Changes:
- Turned yamlSubstituteVariables step into a utils class.
- Added tests
- Adjusted cfManifestSubstituteVariables to use utils class instead of step.
- Adjusted tests
- Adjusted APIs of DebugHelper.

* Re-introduced log statement that shows what variables are being replaced and with what.

* Changing API of YamlUtils to take the script and config as input.

* Test

* Test

* Test

* Test

* Test

* Fixing issue.

* Fixing issue.

* Changes:
- Refactored DebugHelper and YamlUtils to make usage nicer and rely on dependency injection.
- Removed Field for DebugHelper and turned it into local variable.
- Adjusted classes using the above.
- Adjusted tests where necessary.

* Added link to CF standards to YamlUtils also.

* Add docu for step cfManifestSubstituteVariables.md

* Added documentation.

* Added missing script parameter to documentation. Some steps document it, some don't. Right now you need it, so we document it.

* Fixed some layouting and typos

* Beautified exception listing.

* Removed trailing whitespaces to make code climate checks pass.

* Trying to get documentation generated, with all the exceptions to markup one should not use.

* cosmetics.

* cosmetics, part 2

* Code climate changes...

* Inlined deleteFile step.

* Added two more tests to properly check file deletion and output handling.

* Changes:
- adjusted API to take a list of variables files, as does 'cf push --vars-file'
- adjusted API to allow for an optional list of variable key-value-maps as does 'cf push --vars'
- reproduced conflict resolution and overriding behavior of variables files and vars lists
- adjusted tests and documentation

* Added missing paramter to doc comment.

* Re-checked docs for missing paramters or params that have no counterpart in the method signature.

* Adjusted documentation.

* Removed absolute path usage from documentation.

* corrected documentation.

* Changed javadoc comment to plain comment.

* Turned all comments to plain comments.
This commit is contained in:
TheFonz2017
2019-09-06 10:20:35 +02:00
committed by Marcus Holl
parent 211827ac93
commit 41dcebb3a5
20 changed files with 1753 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
documentation/docs/steps/cfManifestSubstituteVariables.md Normal file
View File

@@ -0,0 +1,73 @@
# ${docGenStepName}
## ${docGenDescription}
## ${docGenParameters}
## ${docGenConfiguration}
## ${docJenkinsPluginDependencies}
## Side effects
Unless configured otherwise, this step will *replace* the input `manifest.yml` with a version that has all variable references replaced. This alters the source tree in your Jenkins workspace.
If you prefer to generate a separate output file, use the step's `outputManifestFile` parameter. Keep in mind, however, that your Cloud Foundry deployment step should then also reference this output file - otherwise CF deployment will fail with unresolved variable reference errors.
## Exceptions
* `org.yaml.snakeyaml.scanner.ScannerException` - in case any of the loaded input files contains malformed Yaml and cannot be parsed.
* `hudson.AbortException` - in case of internal errors and when not all variables could be replaced due to missing replacement values.
## Example
Usage of pipeline step:
```groovy
cfManifestSubstituteVariables (
script: this,
manifestFile: "path/to/manifest.yml", //optional, default: manifest.yml
manifestVariablesFiles: ["path/to/manifest-variables.yml"] //optional, default: ['manifest-variables.yml']
manifestVariables: [[key : value], [key : value]] //optional, default: []
)
```
For example, you can refer to the parameters using relative paths (similar to `cf push --vars-file`):
```groovy
cfManifestSubstituteVariables (
script: this,
manifestFile: "manifest.yml",
manifestVariablesFiles: ["manifest-variables.yml"]
)
```
Furthermore, you can also specify variables and their values directly (similar to `cf push --var`):
```groovy
cfManifestSubstituteVariables (
script: this,
manifestFile: "manifest.yml",
manifestVariablesFiles: ["manifest-variables.yml"],
manifestVariables: [[key1 : value1], [key2 : value2]]
)
```
If you are using the Cloud Foundry [Create-Service-Push](https://github.com/dawu415/CF-CLI-Create-Service-Push-Plugin) CLI plugin you will most likely also have a `services-manifest.yml` file.
Also in this file you can specify variable references, that can be resolved from the same variables file, e.g. like this:
```groovy
// resolve variables in manifest.yml
cfManifestSubstituteVariables (
script: this,
manifestFile: "manifest.yml",
manifestVariablesFiles: ["manifest-variables.yml"]
)
// resolve variables in services-manifest.yml from same file.
cfManifestSubstituteVariables (
script: this,
manifestFile: "services-manifest.yml",
manifestVariablesFiles: ["manifest-variables.yml"]
)
```

1
documentation/mkdocs.yml
View File

@@ -9,6 +9,7 @@ nav:
- batsExecuteTests: steps/batsExecuteTests.md
- checkChangeInDevelopment: steps/checkChangeInDevelopment.md
- checksPublishResults: steps/checksPublishResults.md
- cfManifestSubstituteVariables: steps/cfManifestSubstituteVariables.md
- cloudFoundryDeploy: steps/cloudFoundryDeploy.md
- commonPipelineEnvironment: steps/commonPipelineEnvironment.md
- containerExecuteStructureTests: steps/containerExecuteStructureTests.md

53
src/com/sap/piper/variablesubstitution/DebugHelper.groovy Normal file
View File

@@ -0,0 +1,53 @@
package com.sap.piper.variablesubstitution
/**
* Very simple debug helper. Declared as a Field
* and passed the configuration with a call to `setConfig(Map config)`
* in the body of a `call(...)` block, once
* the configuration is available.
*
* If `config.verbose` is set to `true` a message
* issued with `debug(String)` will be logged. Otherwise it will silently be omitted.
*/
class DebugHelper {
/**
* The script which will be used to echo debug messages.
*/
private Script script
/**
* The configuration which will be scanned for a `verbose` flag.
* Only if this is true, will debug messages be written.
*/
private Map config
/**
* Creates a new instance using the given script to issue `echo` commands.
* The given config's `verbose` flag will decide if a message will be logged or not.
* @param script - the script whose `echo` command will be used.
* @param config - the configuration whose `verbose` flag is inspected before logging debug statements.
*/
DebugHelper(Script script, Map config) {
if(!script) {
throw new IllegalArgumentException("[DebugHelper] Script parameter must not be null.")
}
if(!config) {
throw new IllegalArgumentException("[DebugHelper] Config map parameter must not be null.")
}
this.script = script
this.config = config
}
/**
* Logs a debug message if a configuration
* indicates that the `verbose` flag
* is set to `true`
* @param message - the message to log.
*/
void debug(String message) {
if(config.verbose) {
script.echo message
}
}
}

18
src/com/sap/piper/variablesubstitution/ExecutionContext.groovy Normal file
View File

@@ -0,0 +1,18 @@
package com.sap.piper.variablesubstitution
/**
* A simple class to capture context information
* of the execution of yamlSubstituteVariables.
*/
class ExecutionContext {
/**
* Property indicating if the execution
* of yamlSubstituteVariables actually
* substituted any variables at all.
*
* Does NOT indicate that ALL variables were
* actually replaced. If set to true, if just indicates
* that some or all variables have been replaced.
*/
Boolean variablesReplaced = false
}

225
src/com/sap/piper/variablesubstitution/YamlUtils.groovy Normal file
View File

@@ -0,0 +1,225 @@
package com.sap.piper.variablesubstitution
import hudson.AbortException
/**
* A utility class for Yaml data.
* Deals with the substitution of variables within Yaml objects.
*/
class YamlUtils implements Serializable {
private final DebugHelper logger
private final Script script
/**
* Creates a new utils instance for the given script.
* @param script - the script which will be used to call pipeline steps.
* @param logger - an optional debug helper to print debug messages.
*/
YamlUtils(Script script, DebugHelper logger = null) {
if(!script) {
throw new IllegalArgumentException("[YamlUtils] Script must not be null.")
}
this.script = script
this.logger = logger
}
/**
* Substitutes variables references in a given input Yaml object with values that are read from the
* passed variables Yaml object. Variables may be of primitive or complex types.
* The format of variable references follows [Cloud Foundry standards](https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#variable-substitution)
*
* @param inputYaml - the input Yaml data as `Object`. Can be either of type `Map` or `List`.
* @param variablesYaml - the variables Yaml data as `Object`. Can be either of type `Map` or `List` and should
* contain variables names and values to replace variable references contained in `inputYaml`.
* @param context - an `ExecutionContext` that can be used to query whether the script actually replaced any variables.
* @return the YAML object graph of substituted data.
*/
Object substituteVariables(Object inputYaml, Object variablesYaml, ExecutionContext context = null) {
if (!inputYaml) {
throw new IllegalArgumentException("[YamlUtils] Input Yaml data must not be null or empty.")
}
if (!variablesYaml) {
throw new IllegalArgumentException("[YamlUtils] Variables Yaml data must not be null or empty.")
}
return substitute(inputYaml, variablesYaml, context)
}
/**
* Recursively substitutes all variables inside the object tree of the manifest YAML.
* @param manifestNode - the manifest YAML to replace variables in.
* @param variablesData - the variables values.
* @param context - an execution context that can be used to query if any variables were replaced.
* @return a YAML object graph which has all variables replaced.
*/
private Object substitute(Object manifestNode, Object variablesData, ExecutionContext context) {
Map<String, Object> variableSubstitutes = getVariableSubstitutes(variablesData)
if (containsVariableReferences(manifestNode)) {
Object complexResult = null
String stringNode = manifestNode as String
Map<String, String> referencedVariables = getReferencedVariables(stringNode)
referencedVariables.each { referencedVariable ->
String referenceToReplace = referencedVariable.getKey()
String referenceName = referencedVariable.getValue()
Object substitute = variableSubstitutes.get(referenceName)
if (null == substitute) {
logger?.debug("[YamlUtils] WARNING - Found variable reference ${referenceToReplace} in input Yaml but no variable value to replace it with Leaving it unresolved. Check your variables Yaml data and make sure the variable is properly declared.")
return manifestNode
}
script.echo "[YamlUtils] Replacing: ${referenceToReplace} with ${substitute}"
if(isSingleVariableReference(stringNode)) {
logger?.debug("[YamlUtils] Node ${stringNode} is SINGLE variable reference. Substitute type is: ${substitute.getClass().getName()}")
// if the string node we need to do replacements for is
// a reference to a single variable, i.e. should be replaced
// entirely with the variable value, we replace the entire node
// with the variable's value (which can possibly be a complex type).
complexResult = substitute
}
else {
logger?.debug("[YamlUtils] Node ${stringNode} is multi-variable reference or contains additional string constants. Substitute type is: ${substitute.getClass().getName()}")
// if the string node we need to do replacements for contains various
// variable references or a variable reference and constant string additions
// we do a string replacement of the variables inside the node.
String regex = "\\(\\(${referenceName}\\)\\)"
stringNode = stringNode.replaceAll(regex, substitute as String)
}
}
if (context) {
context.variablesReplaced = true // remember that variables were found in the YAML file that have been replaced.
}
return complexResult ?: stringNode
}
else if (manifestNode instanceof List) {
List<Object> listNode = manifestNode as List<Object>
// This copy is only necessary, since Jenkins executes Groovy using
// CPS (https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches)
// and has issues with closures in Java 8 lambda expressions. Otherwise we could replace
// entries of the list in place (using replaceAll(lambdaExpression))
List<Object> copy = new ArrayList<>()
listNode.each { entry ->
copy.add(substitute(entry, variableSubstitutes, context))
}
return copy
}
else if(manifestNode instanceof Map) {
Map<String, Object> mapNode = manifestNode as Map<String, Object>
// This copy is only necessary to avoid immutability errors reported by Jenkins
// runtime environment.
Map<String, Object> copy = new HashMap<>()
mapNode.entrySet().each { entry ->
copy.put(entry.getKey(), substitute(entry.getValue(), variableSubstitutes, context))
}
return copy
}
else {
logger?.debug("[YamlUtils] Found data type ${manifestNode.getClass().getName()} that needs no substitute. Value: ${manifestNode}")
return manifestNode
}
}
/**
* Turns the parsed variables Yaml data into a
* single map. Takes care of multiple YAML sections (separated by ---) if they are found and flattens them into a single
* map if necessary.
* @param variablesYamlData - the variables data as a Yaml object.
* @return the `Map` of variable names mapped to their substitute values.
*/
private Map<String, Object> getVariableSubstitutes(Object variablesYamlData) {
if(variablesYamlData instanceof List) {
return flattenVariablesFileData(variablesYamlData as List)
}
else if (variablesYamlData instanceof Map) {
return variablesYamlData as Map<String, Object>
}
else {
// should never happen (hopefully...)
throw new AbortException("[YamlUtils] Found unsupported data type of variables file after parsing YAML. Expected either List or Map. Got: ${variablesYamlData.getClass().getName()}.")
}
}
/**
* Flattens a list of Yaml sections (which are deemed to be key-value mappings of variable names and values)
* to a single map. In case multiple Yaml sections contain the same key, values will be overridden and the result
* will be undefined.
* @param variablesYamlData - the `List` of Yaml objects of the different sections.
* @return the `Map` of variable substitute mappings.
*/
private Map<String, Object> flattenVariablesFileData(List<Map<String, Object>> variablesYamlData) {
Map<String, Object> substitutes = new HashMap<>()
variablesYamlData.each { map ->
map.entrySet().each { entry ->
substitutes.put(entry.key, entry.value)
}
}
return substitutes
}
/**
* Returns true, if the given object node contains variable references.
* @param node - the object-typed value to check for variable references.
* @return `true`, if this node references at least one variable, `false` otherwise.
*/
private boolean containsVariableReferences(Object node) {
if(!(node instanceof String)) {
// variable references can only be contained in
// string nodes.
return false
}
String stringNode = node as String
return stringNode.contains("((") && stringNode.contains("))")
}
/**
* Returns true, if and only if the entire node passed in as a parameter
* is a variable reference. Returns false if the node references multiple
* variables or if the node embeds the variable reference inside of a constant
* string surrounding, e.g. `This-text-has-((numberOfWords))-words`.
* @param node - the node to check.
* @return `true` if the node is a single variable reference. `false` otherwise.
*/
private boolean isSingleVariableReference(String node) {
// regex matching only if the entire node is a reference. (^ = matches start of word, $ = matches end of word)
String regex = '^\\(\\([\\d\\w-]*\\)\\)$' // use single quote not to have to escape $ (interpolation) sign.
List<String> matches = node.findAll(regex)
return (matches != null && !matches.isEmpty())
}
/**
* Returns a map of variable references (including braces) to plain variable names referenced in the given `String`.
* The keys of the map are the variable references, the values are the names of the referenced variables.
* @param value - the value to look for variable references in.
* @return the `Map` of names of referenced variables.
*/
private Map<String, String> getReferencedVariables(String value) {
Map<String, String> referencesNamesMap = new HashMap<>()
List<String> variableReferences = value.findAll("\\(\\([\\d\\w-]*\\)\\)") // find all variables in braces, e.g. ((my-var_05))
variableReferences.each { reference ->
referencesNamesMap.put(reference, getPlainVariableName(reference))
}
return referencesNamesMap
}
/**
* Expects a variable reference (including braces) as input and returns the plain name
* (by stripping braces) of the variable. E.g. input: `((my_var-04))`, output: `my_var-04`
* @param variableReference - the variable reference including braces.
* @return the plain variable name
*/
private String getPlainVariableName(String variableReference) {
String result = variableReference.replace("((", "")
result = result.replace("))", "")
return result
}
}

600
test/groovy/CfManifestSubstituteVariablesTest.groovy Normal file
View File

@@ -0,0 +1,600 @@
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.rules.ExpectedException
import org.junit.rules.RuleChain
import util.*
import static org.junit.Assert.*
import static util.JenkinsWriteYamlRule.DATA
import static util.JenkinsWriteYamlRule.SERIALIZED_YAML
public class CfManifestSubstituteVariablesTest extends BasePiperTest {
private JenkinsStepRule script = new JenkinsStepRule(this)
private JenkinsReadYamlRule readYamlRule = new JenkinsReadYamlRule(this)
private JenkinsWriteYamlRule writeYamlRule = new JenkinsWriteYamlRule(this)
private JenkinsErrorRule errorRule = new JenkinsErrorRule(this)
private JenkinsEnvironmentRule environmentRule = new JenkinsEnvironmentRule(this)
private JenkinsLoggingRule loggingRule = new JenkinsLoggingRule(this)
private ExpectedException expectedExceptionRule = ExpectedException.none()
private JenkinsFileExistsRule fileExistsRule = new JenkinsFileExistsRule(this, [])
@Rule
public RuleChain rules = Rules
.getCommonRules(this)
.around(fileExistsRule)
.around(readYamlRule)
.around(writeYamlRule)
.around(errorRule)
.around(environmentRule)
.around(loggingRule)
.around(script)
.around(expectedExceptionRule)
@Before
public void setup() {
readYamlRule.registerYaml("manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest.yml")))
.registerYaml("manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest-variables.yml")))
.registerYaml("test/resources/variableSubstitution/manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest.yml")))
.registerYaml("test/resources/variableSubstitution/manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest-variables.yml")))
.registerYaml("test/resources/variableSubstitution/invalid_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/invalid_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/novars_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/novars_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/multi_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/multi_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/datatypes_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/datatypes_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/datatypes_manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/datatypes_manifest-variables.yml")))
.registerYaml("test/resources/variableSubstitution/manifest-variables-conflicting.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest-variables-conflicting.yml")))
}
@Test
public void substituteVariables_SkipsExecution_If_ManifestNotPresent() throws Exception {
String manifestFileName = "nonexistent/manifest.yml"
String variablesFileName = "nonexistent/manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Could not find YAML file at ${manifestFileName}. Skipping variable substitution.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_SkipsExecution_If_VariablesFileNotPresent() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "nonexistent/manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Did not find manifest variable substitution file at ${variablesFileName}.")
expectedExceptionRule.expect(hudson.AbortException)
expectedExceptionRule.expectMessage("[CFManifestSubstituteVariables] Could not find all given manifest variable substitution files. Make sure all files given as manifestVariablesFiles exist.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_Throws_If_manifestInvalid() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
//check that exception is thrown and that it has the correct message.
expectedExceptionRule.expect(org.yaml.snakeyaml.scanner.ScannerException)
expectedExceptionRule.expectMessage("found character '%' that cannot start any token. (Do not use % for indentation)")
loggingRule.expect("[CFManifestSubstituteVariables] Could not load manifest file at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_Throws_If_manifestVariablesInvalid() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
//check that exception is thrown and that it has the correct message.
expectedExceptionRule.expect(org.yaml.snakeyaml.scanner.ScannerException)
expectedExceptionRule.expectMessage("found character '%' that cannot start any token. (Do not use % for indentation)")
loggingRule.expect("[CFManifestSubstituteVariables] Could not load manifest variables file at ${variablesFileName}")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_UsesDefaultFileName_If_NoManifestSpecified() throws Exception {
// In this test, we check that the implementation will resort to the default manifest file name.
// Since the file is not present, the implementation should stop, but the log should indicate that the
// the default file name was used.
String manifestFileName = "manifest.yml" // default name should be chosen.
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Could not find YAML file at ${manifestFileName}. Skipping variable substitution.")
// execute step
script.step.cfManifestSubstituteVariables script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_UsesDefaultFileName_If_NoVariablesFileSpecified() throws Exception {
// In this test, we check that the implementation will resort to the default manifest _variables_ file name.
// Since the file is not present, the implementation should stop, but the log should indicate that the
// the default file name was used.
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "manifest-variables.yml" // default file name that should be chosen.
fileExistsRule.registerExistingFile(manifestFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Did not find manifest variable substitution file at ${variablesFileName}.")
.expect("[CFManifestSubstituteVariables] Could not find any default manifest variable substitution file at ${variablesFileName}, and no manifest variables list was specified. Skipping variable substitution.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_ReplacesVariablesProperly_InSingleYamlFiles() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
Map<String, Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that there are no unresolved variables left.
assertAllVariablesReplaced(yamlStringAfterReplacement)
// check that resolved variables have expected values
assertCorrectVariableResolution(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
private void assertAllVariablesReplaced(String yamlStringAfterReplacement) {
assertFalse(yamlStringAfterReplacement.contains("(("))
assertFalse(yamlStringAfterReplacement.contains("))"))
}
private void assertCorrectVariableResolution(Map<String, Object> manifestDataAfterReplacement) {
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("name").equals("uniquePrefix-catalog-service-odatav2-0.0.1"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("routes").get(0).get("route").equals("uniquePrefix-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(0).equals("uniquePrefix-catalog-service-odatav2-xsuaa"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(1).equals("uniquePrefix-catalog-service-odatav2-hana"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("xsuaa-instance-name").equals("uniquePrefix-catalog-service-odatav2-xsuaa"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("db_service_instance_name").equals("uniquePrefix-catalog-service-odatav2-hana"))
}
@Test
public void substituteVariables_ReplacesVariablesProperly_InMultiYamlFiles() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/multi_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
List<Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that there are no unresolved variables left.
assertAllVariablesReplaced(yamlStringAfterReplacement)
//check that result still is a multi-YAML file.
assertEquals("Dumped YAML after replacement should still be a multi-YAML file.",2, manifestDataAfterReplacement.size())
// check that resolved variables have expected values
manifestDataAfterReplacement.each { yaml ->
assertCorrectVariableResolution(yaml as Map<String, Object>)
}
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_ReplacesVariablesFromListProperly_whenNoManifestVariablesFilesGiven_InMultiYamlFiles() throws Exception {
// This test replaces variables in multi-yaml files from a list of specified variables
// the the user has not specified any variable substitution files list.
String manifestFileName = "test/resources/variableSubstitution/multi_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
List<Map<String, Object>> variablesList = [
["unique-prefix" : "uniquePrefix"],
["xsuaa-instance-name" : "uniquePrefix-catalog-service-odatav2-xsuaa"],
["hana-instance-name" : "uniquePrefix-catalog-service-odatav2-hana"]
]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
//.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariables: variablesList, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
List<Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that there are no unresolved variables left.
assertAllVariablesReplaced(yamlStringAfterReplacement)
//check that result still is a multi-YAML file.
assertEquals("Dumped YAML after replacement should still be a multi-YAML file.",2, manifestDataAfterReplacement.size())
// check that resolved variables have expected values
manifestDataAfterReplacement.each { yaml ->
assertCorrectVariableResolution(yaml as Map<String, Object>)
}
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_ReplacesVariablesFromListProperly_whenEmptyManifestVariablesFilesGiven_InMultiYamlFiles() throws Exception {
// This test replaces variables in multi-yaml files from a list of specified variables
// the the user has specified an empty list of variable substitution files.
String manifestFileName = "test/resources/variableSubstitution/multi_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
List<String> variablesFiles = []
List<Map<String, Object>> variablesList = [
["unique-prefix" : "uniquePrefix"],
["xsuaa-instance-name" : "uniquePrefix-catalog-service-odatav2-xsuaa"],
["hana-instance-name" : "uniquePrefix-catalog-service-odatav2-hana"]
]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
//.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, manifestVariables: variablesList, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
List<Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that there are no unresolved variables left.
assertAllVariablesReplaced(yamlStringAfterReplacement)
//check that result still is a multi-YAML file.
assertEquals("Dumped YAML after replacement should still be a multi-YAML file.",2, manifestDataAfterReplacement.size())
// check that resolved variables have expected values
manifestDataAfterReplacement.each { yaml ->
assertCorrectVariableResolution(yaml as Map<String, Object>)
}
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_SkipsExecution_If_NoVariablesInManifest() throws Exception {
// This test makes sure that, if no variables are found in a manifest that need
// to be replaced, the execution is eventually skipped and the manifest remains
// untouched.
String manifestFileName = "test/resources/variableSubstitution/novars_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] No variables were found or could be replaced in ${manifestFileName}. Skipping variable substitution.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
//Check that nothing was written
assertNull(writeYamlRule.files[manifestFileName])
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_SupportsAllDataTypes() throws Exception {
// This test makes sure that, all datatypes supported by YAML are also
// properly substituted by the substituteVariables step.
// In particular this includes variables of type:
// Integer, Boolean, String, Float and inline JSON documents (which are parsed as multi-line strings)
// and complex types (like other YAML objects).
// The test also checks the differing behaviour when substituting nodes that only consist of a
// variable reference and nodes that contains several variable references or additional string constants.
String manifestFileName = "test/resources/variableSubstitution/datatypes_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/datatypes_manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
Map<String, Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
assertAllVariablesReplaced(yamlStringAfterReplacement)
assertCorrectVariableResolution(manifestDataAfterReplacement)
assertDataTypeAndSubstitutionCorrectness(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
private void assertDataTypeAndSubstitutionCorrectness(Map<String, Object> manifestDataAfterReplacement) {
// See datatypes_manifest.yml and datatypes_manifest-variables.yml.
// Note: For debugging consider turning on YAML writing to a file in JenkinsWriteYamlRule to see the
// actual outcome of replacing variables (for visual inspection).
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("instances").equals(1))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("instances") instanceof Integer)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(0) instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("booleanVariable").equals(true))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("booleanVariable") instanceof Boolean)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("floatVariable") == 0.25)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("floatVariable") instanceof Double)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("json-variable") instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("object-variable") instanceof Map)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("string-variable").startsWith("true-0.25-1-"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("string-variable") instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("single-var-with-string-constants").equals("true-with-some-more-text"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("single-var-with-string-constants") instanceof String)
}
@Test
public void substituteVariables_replacesManifestIfNoOutputGiven() throws Exception {
// Test that makes sure that the original input manifest file is replaced with
// a version that has variables replaced (by deleting the original file and
// dumping a new one with the same name).
String manifestFileName = "test/resources/variableSubstitution/datatypes_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/datatypes_manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Successfully deleted file '${manifestFileName}'.")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
Map<String, Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
assertAllVariablesReplaced(yamlStringAfterReplacement)
assertCorrectVariableResolution(manifestDataAfterReplacement)
assertDataTypeAndSubstitutionCorrectness(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_writesToOutputFileIfGiven() throws Exception {
// Test that makes sure that the output is written to the specified file and that the original input manifest
// file is NOT deleted / replaced.
String manifestFileName = "test/resources/variableSubstitution/datatypes_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/datatypes_manifest-variables.yml"
List<String> variablesFiles = [ variablesFileName ]
String outputFileName = "output.yml"
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${outputFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, outputManifestFile: outputFileName, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[outputFileName].get(SERIALIZED_YAML) as String
Map<String, Object> manifestDataAfterReplacement = writeYamlRule.files[outputFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// make sure the input file was NOT deleted.
assertFalse(loggingRule.expected.contains("[CFManifestSubstituteVariables] Successfully deleted file '${manifestFileName}'."))
assertAllVariablesReplaced(yamlStringAfterReplacement)
assertCorrectVariableResolution(manifestDataAfterReplacement)
assertDataTypeAndSubstitutionCorrectness(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_resolvesConflictsProperly_InMultiYamlFiles() throws Exception {
// This test checks if the resolution and of conflicting variables and the
// overriding nature of variable lists vs. variable files lists is working properly.
String manifestFileName = "test/resources/variableSubstitution/multi_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
String conflictingVariablesFileName = "test/resources/variableSubstitution/manifest-variables-conflicting.yml"
List<String> variablesFiles = [ variablesFileName, conflictingVariablesFileName ] //introducing a conflicting file whose entries should win, since it is last in the list
List<Map<String, Object>> variablesList = [
["unique-prefix" : "uniquePrefix-from-vars-list"],
["unique-prefix" : "uniquePrefix-from-vars-list-conflicting"] // introduce a conflict that should win, since it is last in the list.
]
fileExistsRule.registerExistingFile(manifestFileName)
fileExistsRule.registerExistingFile(variablesFileName)
fileExistsRule.registerExistingFile(conflictingVariablesFileName)
// check that a proper log is written.
loggingRule.expect("[CFManifestSubstituteVariables] Loaded manifest at ${manifestFileName}!")
//.expect("[CFManifestSubstituteVariables] Loaded variables file at ${variablesFileName}!")
.expect("[CFManifestSubstituteVariables] Replaced variables in ${manifestFileName}.")
.expect("[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${manifestFileName}.")
// execute step
script.step.cfManifestSubstituteVariables manifestFile: manifestFileName, manifestVariablesFiles: variablesFiles, manifestVariables: variablesList, script: nullScript
String yamlStringAfterReplacement = writeYamlRule.files[manifestFileName].get(SERIALIZED_YAML) as String
List<Object> manifestDataAfterReplacement = writeYamlRule.files[manifestFileName].get(DATA)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that there are no unresolved variables left.
assertAllVariablesReplaced(yamlStringAfterReplacement)
//check that result still is a multi-YAML file.
assertEquals("Dumped YAML after replacement should still be a multi-YAML file.",2, manifestDataAfterReplacement.size())
// check that resolved variables have expected values
manifestDataAfterReplacement.each { yaml ->
assertCorrectVariableSubstitutionUnderConflictAndWithOverriding(yaml as Map<String, Object>)
}
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
private void assertCorrectVariableSubstitutionUnderConflictAndWithOverriding(Map<String, Object> manifestDataAfterReplacement) {
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("name").equals("uniquePrefix-from-vars-list-conflicting-catalog-service-odatav2-0.0.1"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("routes").get(0).get("route").equals("uniquePrefix-from-vars-list-conflicting-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(0).equals("uniquePrefix-catalog-service-odatav2-xsuaa-conflicting-from-file"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(1).equals("uniquePrefix-catalog-service-odatav2-hana"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("xsuaa-instance-name").equals("uniquePrefix-catalog-service-odatav2-xsuaa-conflicting-from-file"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("db_service_instance_name").equals("uniquePrefix-catalog-service-odatav2-hana"))
}
}

271
test/groovy/com/sap/piper/variablesubstitution/YamlUtilsTest.groovy Normal file
View File

@@ -0,0 +1,271 @@
package com.sap.piper.variablesubstitution
import org.junit.Before
import static org.junit.Assert.*
import org.junit.Rule
import org.junit.Test
import org.junit.rules.ExpectedException;
import org.junit.rules.RuleChain;
import util.BasePiperTest
import util.JenkinsEnvironmentRule
import util.JenkinsErrorRule
import util.JenkinsLoggingRule
import util.JenkinsReadYamlRule
import util.JenkinsWriteYamlRule
import util.Rules
class YamlUtilsTest extends BasePiperTest {
private JenkinsReadYamlRule readYamlRule = new JenkinsReadYamlRule(this)
private JenkinsWriteYamlRule writeYamlRule = new JenkinsWriteYamlRule(this)
private JenkinsErrorRule errorRule = new JenkinsErrorRule(this)
private JenkinsEnvironmentRule environmentRule = new JenkinsEnvironmentRule(this)
private JenkinsLoggingRule loggingRule = new JenkinsLoggingRule(this)
private ExpectedException expectedExceptionRule = ExpectedException.none()
private YamlUtils yamlUtils
@Rule
public RuleChain rules = Rules
.getCommonRules(this)
.around(readYamlRule)
.around(writeYamlRule)
.around(errorRule)
.around(environmentRule)
.around(loggingRule)
.around(expectedExceptionRule)
@Before
public void setup() {
yamlUtils = new YamlUtils(nullScript)
readYamlRule.registerYaml("manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest.yml")))
.registerYaml("manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest-variables.yml")))
.registerYaml("test/resources/variableSubstitution/manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest.yml")))
.registerYaml("test/resources/variableSubstitution/manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/manifest-variables.yml")))
.registerYaml("test/resources/variableSubstitution/invalid_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/invalid_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/novars_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/novars_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/multi_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/multi_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/datatypes_manifest.yml", new FileInputStream(new File("test/resources/variableSubstitution/datatypes_manifest.yml")))
.registerYaml("test/resources/variableSubstitution/datatypes_manifest-variables.yml", new FileInputStream(new File("test/resources/variableSubstitution/datatypes_manifest-variables.yml")))
}
@Test
public void substituteVariables_Fails_If_InputYamlIsNullOrEmpty() throws Exception {
expectedExceptionRule.expect(IllegalArgumentException)
expectedExceptionRule.expectMessage("[YamlUtils] Input Yaml data must not be null or empty.")
yamlUtils.substituteVariables(null, null)
}
@Test
public void substituteVariables_Fails_If_VariablesYamlIsNullOrEmpty() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
expectedExceptionRule.expect(IllegalArgumentException)
expectedExceptionRule.expectMessage("[YamlUtils] Variables Yaml data must not be null or empty.")
Object input = nullScript.readYaml file: manifestFileName
// execute step
yamlUtils.substituteVariables(input, null)
}
@Test
public void substituteVariables_Throws_If_InputYamlIsInvalid() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
//check that exception is thrown and that it has the correct message.
expectedExceptionRule.expect(org.yaml.snakeyaml.scanner.ScannerException)
expectedExceptionRule.expectMessage("found character '%' that cannot start any token. (Do not use % for indentation)")
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
yamlUtils.substituteVariables(input, variables)
}
@Test
public void substituteVariables_Throws_If_VariablesYamlInvalid() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/invalid_manifest.yml"
//check that exception is thrown and that it has the correct message.
expectedExceptionRule.expect(org.yaml.snakeyaml.scanner.ScannerException)
expectedExceptionRule.expectMessage("found character '%' that cannot start any token. (Do not use % for indentation)")
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
yamlUtils.substituteVariables(input, variables)
}
@Test
public void substituteVariables_ReplacesVariablesProperly_InSingleYamlFiles() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
Map<String, Object> manifestDataAfterReplacement = yamlUtils.substituteVariables(input, variables)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that resolved variables have expected values
assertCorrectVariableResolution(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
private void assertAllVariablesReplaced(String yamlStringAfterReplacement) {
assertFalse(yamlStringAfterReplacement.contains("(("))
assertFalse(yamlStringAfterReplacement.contains("))"))
}
private void assertCorrectVariableResolution(Map<String, Object> manifestDataAfterReplacement) {
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("name").equals("uniquePrefix-catalog-service-odatav2-0.0.1"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("routes").get(0).get("route").equals("uniquePrefix-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(0).equals("uniquePrefix-catalog-service-odatav2-xsuaa"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(1).equals("uniquePrefix-catalog-service-odatav2-hana"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("xsuaa-instance-name").equals("uniquePrefix-catalog-service-odatav2-xsuaa"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("db_service_instance_name").equals("uniquePrefix-catalog-service-odatav2-hana"))
}
@Test
public void substituteVariables_ReplacesVariablesProperly_InMultiYamlData() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/multi_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
List<Object> manifestDataAfterReplacement = yamlUtils.substituteVariables(input, variables)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
//check that result still is a multi-YAML file.
assertEquals("Dumped YAML after replacement should still be a multi-YAML file.",2, manifestDataAfterReplacement.size())
// check that resolved variables have expected values
manifestDataAfterReplacement.each { yaml ->
assertCorrectVariableResolution(yaml as Map<String, Object>)
}
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_ReturnsOriginalIfNoVariablesPresent() throws Exception {
// This test makes sure that, if no variables are found in a manifest that need
// to be replaced, the execution is eventually skipped and the manifest remains
// untouched.
String manifestFileName = "test/resources/variableSubstitution/novars_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
ExecutionContext context = new ExecutionContext()
Object result = yamlUtils.substituteVariables(input, variables, context)
//Check that nothing was written
assertNotNull(result)
assertFalse(context.variablesReplaced)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
@Test
public void substituteVariables_SupportsAllDataTypes() throws Exception {
// This test makes sure that, all datatypes supported by YAML are also
// properly substituted by the substituteVariables step.
// In particular this includes variables of type:
// Integer, Boolean, String, Float and inline JSON documents (which are parsed as multi-line strings)
// and complex types (like other YAML objects).
// The test also checks the differing behaviour when substituting nodes that only consist of a
// variable reference and nodes that contains several variable references or additional string constants.
String manifestFileName = "test/resources/variableSubstitution/datatypes_manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/datatypes_manifest-variables.yml"
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
ExecutionContext context = new ExecutionContext()
Map<String, Object> manifestDataAfterReplacement = yamlUtils.substituteVariables(input, variables, context)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
assertCorrectVariableResolution(manifestDataAfterReplacement)
assertDataTypeAndSubstitutionCorrectness(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
private void assertDataTypeAndSubstitutionCorrectness(Map<String, Object> manifestDataAfterReplacement) {
// See datatypes_manifest.yml and datatypes_manifest-variables.yml.
// Note: For debugging consider turning on YAML writing to a file in JenkinsWriteYamlRule to see the
// actual outcome of replacing variables (for visual inspection).
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("instances").equals(1))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("instances") instanceof Integer)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("services").get(0) instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("booleanVariable").equals(true))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("booleanVariable") instanceof Boolean)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("floatVariable") == 0.25)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("floatVariable") instanceof Double)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("json-variable") instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("object-variable") instanceof Map)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("string-variable").startsWith("true-0.25-1-"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("string-variable") instanceof String)
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("single-var-with-string-constants").equals("true-with-some-more-text"))
assertTrue(manifestDataAfterReplacement.get("applications").get(0).get("env").get("single-var-with-string-constants") instanceof String)
}
@Test
public void substituteVariables_DoesNotFail_If_ExecutionContextIsNull() throws Exception {
String manifestFileName = "test/resources/variableSubstitution/manifest.yml"
String variablesFileName = "test/resources/variableSubstitution/manifest-variables.yml"
Object input = nullScript.readYaml file: manifestFileName
Object variables = nullScript.readYaml file: variablesFileName
// execute step
Map<String, Object> manifestDataAfterReplacement = yamlUtils.substituteVariables(input, variables, null)
//Check that something was written
assertNotNull(manifestDataAfterReplacement)
// check that resolved variables have expected values
assertCorrectVariableResolution(manifestDataAfterReplacement)
// check that the step was marked as a success (even if it did do nothing).
assertJobStatusSuccess()
}
}

21
test/groovy/util/JenkinsFileExistsRule.groovy
View File

@@ -10,11 +10,21 @@ class JenkinsFileExistsRule implements TestRule {
final BasePipelineTest testInstance
final List existingFiles
/**
* The List of files that have been queried via `fileExists`
*/
final List queriedFiles = []
JenkinsFileExistsRule(BasePipelineTest testInstance, List existingFiles) {
this.testInstance = testInstance
this.existingFiles = existingFiles
}
JenkinsFileExistsRule registerExistingFile(String file) {
existingFiles.add(file)
return this
}
@Override
Statement apply(Statement base, Description description) {
return statement(base)
@@ -25,8 +35,15 @@ class JenkinsFileExistsRule implements TestRule {
@Override
void evaluate() throws Throwable {
testInstance.helper.registerAllowedMethod('fileExists', [String.class], {s -> return s in existingFiles})
testInstance.helper.registerAllowedMethod('fileExists', [Map.class], {m -> return m.file in existingFiles})
testInstance.helper.registerAllowedMethod('fileExists', [String.class], {s ->
queriedFiles.add(s)
return s in existingFiles
})
testInstance.helper.registerAllowedMethod('fileExists', [Map.class], {m ->
queriedFiles.add(m.file)
return m.file in existingFiles}
)
base.evaluate()
}

3
test/groovy/util/JenkinsLoggingRule.groovy
View File

@@ -24,8 +24,9 @@ class JenkinsLoggingRule implements TestRule {
this.testInstance = testInstance
}
public void expect(String substring) {
public JenkinsLoggingRule expect(String substring) {
expected.add(substring)
return this
}
@Override

28
test/groovy/util/JenkinsReadYamlRule.groovy
View File

@@ -42,11 +42,37 @@ class JenkinsReadYamlRule implements TestRule {
} else {
throw new IllegalArgumentException("Key 'text' and 'file' are both missing in map ${m}.")
}
return new Yaml().load(yml)
return readYaml(yml)
})
base.evaluate()
}
}
}
/**
* Mimicking code of the original library (link below).
* <p>
* Yaml files may contain several YAML sections, separated by ---.
* This loads them all and returns a {@code List} of entries in case multiple sections were found or just
* a single {@code Object}, if only one section was read.
* @see https://github.com/jenkinsci/pipeline-utility-steps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/ReadYamlStep.java
*/
private def readYaml(def yml) {
Iterable<Object> yaml = new Yaml().loadAll(yml)
List<Object> result = new LinkedList<Object>()
for (Object data : yaml) {
result.add(data)
}
// If only one YAML document, return it directly
if (result.size() == 1) {
return result.get(0);
}
return result;
}
}

59
test/groovy/util/JenkinsWriteYamlRule.groovy Normal file
View File

@@ -0,0 +1,59 @@
package util
import com.lesfurets.jenkins.unit.BasePipelineTest
import org.yaml.snakeyaml.Yaml
import static org.junit.Assert.*
import org.junit.rules.TestRule
import org.junit.runner.Description
import org.junit.runners.model.Statement
class JenkinsWriteYamlRule implements TestRule {
final BasePipelineTest testInstance
static final String DATA = "DATA" // key in files map to retrieve Yaml object graph data..
static final String CHARSET = "CHARSET" // key in files map to retrieve the charset of the serialized Yaml.
static final String SERIALIZED_YAML = "SERIALIZED_YAML" // key in files map to retrieve serialized Yaml.
Map<String, Map<String, Object>> files = new HashMap<>()
JenkinsWriteYamlRule(BasePipelineTest testInstance) {
this.testInstance = testInstance
}
@Override
Statement apply(Statement base, Description description) {
return statement(base)
}
private Statement statement(final Statement base) {
return new Statement() {
@Override
void evaluate() throws Throwable {
testInstance.helper.registerAllowedMethod( 'writeYaml', [Map], { parameterMap ->
assertNotNull(parameterMap.file)
assertNotNull(parameterMap.data)
// charset is optional.
Yaml yaml = new Yaml()
StringWriter writer = new StringWriter()
yaml.dump(parameterMap.data, writer)
// Enable this to actually produce a file.
// yaml.dump(parameterMap.data, new FileWriter(parameterMap.file))
// yaml.dump(parameterMap.data, new FileWriter("test/resources/variableSubstitution/manifest_out.yml"))
Map<String, Object> details = new HashMap<>()
details.put(DATA, parameterMap.data)
details.put(CHARSET, parameterMap.charset ?: "UTF-8")
details.put(SERIALIZED_YAML, writer.toString())
files[parameterMap.file] = details
})
base.evaluate()
}
}
}
}

19
test/resources/variableSubstitution/datatypes_manifest-variables.yml Normal file
View File

@@ -0,0 +1,19 @@
---
unique-prefix: uniquePrefix # A unique prefix. E.g. your D/I/C-User
xsuaa-instance-name: uniquePrefix-catalog-service-odatav2-xsuaa
hana-instance-name: uniquePrefix-catalog-service-odatav2-hana
integer-variable: 1
boolean-variable: Yes
float-variable: 0.25
json-variable: >
[
{"name":"token-destination",
"url":"https://www.google.com",
"forwardAuthToken": true}
]
object-variable:
hello: "world"
this: "is an object with"
one: 1
float: 25.0
bool: Yes

27
test/resources/variableSubstitution/datatypes_manifest.yml Normal file
View File

@@ -0,0 +1,27 @@
---
applications:
- name: ((unique-prefix))-catalog-service-odatav2-0.0.1
memory: 1024M
disk_quota: 512M
instances: ((integer-variable))
buildpacks:
- java_buildpack
path: ./srv/target/srv-backend-0.0.1-SNAPSHOT.jar
routes:
- route: ((unique-prefix))-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com
services:
- ((xsuaa-instance-name)) # requires an instance of xsuaa instantiated with xs-security.json of this project. See services-manifest.yml.
- ((hana-instance-name)) # requires an instance of hana service with plan hdi-shared. See services-manifest.yml.
env:
spring.profiles.active: cloud # activate the spring profile named 'cloud'.
xsuaa-instance-name: ((xsuaa-instance-name))
db_service_instance_name: ((hana-instance-name))
booleanVariable: ((boolean-variable))
floatVariable: ((float-variable))
json-variable: ((json-variable))
object-variable: ((object-variable))
string-variable: ((boolean-variable))-((float-variable))-((integer-variable))-((json-variable))
single-var-with-string-constants: ((boolean-variable))-with-some-more-text

2
test/resources/variableSubstitution/invalid_manifest.yml Normal file
View File

@@ -0,0 +1,2 @@
---
test: %invalid

4
test/resources/variableSubstitution/manifest-variables-conflicting.yml Normal file
View File

@@ -0,0 +1,4 @@
---
unique-prefix: uniquePrefix-conflicting-from-file # A unique prefix. E.g. your D/I/C-User
xsuaa-instance-name: uniquePrefix-catalog-service-odatav2-xsuaa-conflicting-from-file

4
test/resources/variableSubstitution/manifest-variables.yml Normal file
View File

@@ -0,0 +1,4 @@
---
unique-prefix: uniquePrefix # A unique prefix. E.g. your D/I/C-User
xsuaa-instance-name: uniquePrefix-catalog-service-odatav2-xsuaa
hana-instance-name: uniquePrefix-catalog-service-odatav2-hana

21
test/resources/variableSubstitution/manifest.yml Normal file
View File

@@ -0,0 +1,21 @@
---
applications:
- name: ((unique-prefix))-catalog-service-odatav2-0.0.1
memory: 1024M
disk_quota: 512M
instances: 1
buildpacks:
- java_buildpack
path: ./srv/target/srv-backend-0.0.1-SNAPSHOT.jar
routes:
- route: ((unique-prefix))-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com
services:
- ((xsuaa-instance-name)) # requires an instance of xsuaa instantiated with xs-security.json of this project. See services-manifest.yml.
- ((hana-instance-name)) # requires an instance of hana service with plan hdi-shared. See services-manifest.yml.
env:
spring.profiles.active: cloud # activate the spring profile named 'cloud'.
xsuaa-instance-name: ((xsuaa-instance-name))
db_service_instance_name: ((hana-instance-name))

43
test/resources/variableSubstitution/multi_manifest.yml Normal file
View File

@@ -0,0 +1,43 @@
---
applications:
- name: ((unique-prefix))-catalog-service-odatav2-0.0.1
memory: 1024M
disk_quota: 512M
instances: 1
buildpacks:
- java_buildpack
path: ./srv/target/srv-backend-0.0.1-SNAPSHOT.jar
routes:
- route: ((unique-prefix))-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com
services:
- ((xsuaa-instance-name)) # requires an instance of xsuaa instantiated with xs-security.json of this project. See services-manifest.yml.
- ((hana-instance-name)) # requires an instance of hana service with plan hdi-shared. See services-manifest.yml.
env:
spring.profiles.active: cloud # activate the spring profile named 'cloud'.
xsuaa-instance-name: ((xsuaa-instance-name))
db_service_instance_name: ((hana-instance-name))
---
applications:
- name: ((unique-prefix))-catalog-service-odatav2-0.0.1
memory: 1024M
disk_quota: 512M
instances: 1
buildpacks:
- java_buildpack
path: ./srv/target/srv-backend-0.0.1-SNAPSHOT.jar
routes:
- route: ((unique-prefix))-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com
services:
- ((xsuaa-instance-name)) # requires an instance of xsuaa instantiated with xs-security.json of this project. See services-manifest.yml.
- ((hana-instance-name)) # requires an instance of hana service with plan hdi-shared. See services-manifest.yml.
env:
spring.profiles.active: cloud # activate the spring profile named 'cloud'.
xsuaa-instance-name: ((xsuaa-instance-name))
db_service_instance_name: ((hana-instance-name))

21
test/resources/variableSubstitution/novars_manifest.yml Normal file
View File

@@ -0,0 +1,21 @@
---
applications:
- name: test-catalog-service-odatav2-0.0.1
memory: 1024M
disk_quota: 512M
instances: 1
buildpacks:
- java_buildpack
path: ./srv/target/srv-backend-0.0.1-SNAPSHOT.jar
routes:
- route: test-catalog-service-odatav2-001.cfapps.eu10.hana.ondemand.com
services:
- xsuaa-instance-name # requires an instance of xsuaa instantiated with xs-security.json of this project. See services-manifest.yml.
- hana-instance-name # requires an instance of hana service with plan hdi-shared. See services-manifest.yml.
env:
spring.profiles.active: cloud # activate the spring profile named 'cloud'.
xsuaa-instance-name: xsuaa-instance-name
db_service_instance_name: hana-instance-name

264
vars/cfManifestSubstituteVariables.groovy Normal file
View File

@@ -0,0 +1,264 @@
import com.sap.piper.ConfigurationHelper
import com.sap.piper.GenerateDocumentation
import com.sap.piper.variablesubstitution.ExecutionContext
import com.sap.piper.variablesubstitution.DebugHelper
import com.sap.piper.variablesubstitution.YamlUtils
import groovy.transform.Field
import static com.sap.piper.Prerequisites.checkScript
@Field String STEP_NAME = getClass().getName()
@Field Set GENERAL_CONFIG_KEYS = []
@Field Set STEP_CONFIG_KEYS = GENERAL_CONFIG_KEYS + [
/**
* The `String` path of the Yaml file to replace variables in.
* Defaults to "manifest.yml" if not specified otherwise.
*/
'manifestFile',
/**
* The `String` path of the Yaml file to produce as output.
* If not specified this will default to `manifestFile` and overwrite it.
*/
'outputManifestFile',
/**
* The `List` of `String` paths of the Yaml files containing the variable values to use as a replacement in the manifest file.
* Defaults to `["manifest-variables.yml"]` if not specified otherwise. The order of the files given in the list is relevant
* in case there are conflicting variable names and values within variable files. In such a case, the values of the last file win.
*/
'manifestVariablesFiles',
/**
* A `List` of `Map` entries for key-value pairs used for variable substitution within the file given by `manifestFile`.
* Defaults to an empty list, if not specified otherwise. This can be used to set variables like it is provided
* by `cf push --var key=value`.
*
* The order of the maps of variables given in the list is relevant in case there are conflicting variable names and values
* between maps contained within the list. In case of conflicts, the last specified map in the list will win.
*
* Though each map entry in the list can contain more than one key-value pair for variable substitution, it is recommended
* to stick to one entry per map, and rather declare more maps within the list. The reason is that
* if a map in the list contains more than one key-value entry, and the entries are conflicting, the
* conflict resolution behavior is undefined (since map entries have no sequence).
*
* Note: variables defined via `manifestVariables` always win over conflicting variables defined via any file given
* by `manifestVariablesFiles` - no matter what is declared before. This reproduces the same behavior as can be
* observed when using `cf push --var` in combination with `cf push --vars-file`.
*/
'manifestVariables'
]
@Field Set PARAMETER_KEYS = STEP_CONFIG_KEYS
/*
* Step to substitute variables in a given YAML file with those specified in one or more variables files given by the
* `manifestVariablesFiles` parameter. This follows the behavior of `cf push --vars-file`, and can be
* used as a pre-deployment step if commands other than `cf push` are used for deployment (e.g. `cf blue-green-deploy`).
*
* The format to reference a variable in the manifest YAML file is to use double parentheses `((` and `))`, e.g. `((variableName))`.
*
* You can declare variable assignments as key value-pairs inside a YAML variables file following the
* [Cloud Foundry standards](https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#variable-substitution) format.
*
* Optionally, you can also specify a direct list of key-value mappings for variables using the `manifestVariables` parameter.
* Variables given in the `manifestVariables` list will take precedence over those found in variables files. This follows
* the behavior of `cf push --var`, and works in combination with `manifestVariablesFiles`.
*
* The step is activated by the presence of the file specified by the `manifestFile` parameter and all variables files
* specified by the `manifestVariablesFiles` parameter, or if variables are passed in directly via `manifestVariables`.
*
* In case no `manifestVariablesFiles` were explicitly specified, a default named `manifest-variables.yml` will be looked
* for and if present will activate this step also. This is to support convention over configuration.
*/
@GenerateDocumentation
void call(Map arguments = [:]) {
handlePipelineStepErrors (stepName: STEP_NAME, stepParameters: arguments) {
def script = checkScript(this, arguments) ?: this
// load default & individual configuration
Map config = ConfigurationHelper.newInstance(this)
.loadStepDefaults()
.mixinStepConfig(script.commonPipelineEnvironment, STEP_CONFIG_KEYS)
.mixinStageConfig(script.commonPipelineEnvironment, arguments.stageName ?: env.STAGE_NAME, STEP_CONFIG_KEYS)
.mixin(arguments, PARAMETER_KEYS)
.use()
String defaultManifestFileName = "manifest.yml"
String defaultManifestVariablesFileName = "manifest-variables.yml"
Boolean manifestVariablesFilesExplicitlySpecified = config.manifestVariablesFiles != null
String manifestFilePath = config.manifestFile ?: defaultManifestFileName
List<String> manifestVariablesFiles = (config.manifestVariablesFiles != null) ? config.manifestVariablesFiles : [ defaultManifestVariablesFileName ]
List<Map<String, Object>> manifestVariablesList = config.manifestVariables ?: []
String outputFilePath = config.outputManifestFile ?: manifestFilePath
DebugHelper debugHelper = new DebugHelper(script, config)
YamlUtils yamlUtils = new YamlUtils(script, debugHelper)
Boolean manifestExists = fileExists manifestFilePath
Boolean manifestVariablesFilesExist = allManifestVariableFilesExist(manifestVariablesFiles)
Boolean manifestVariablesListSpecified = !manifestVariablesList.isEmpty()
if (!manifestExists) {
echo "[CFManifestSubstituteVariables] Could not find YAML file at ${manifestFilePath}. Skipping variable substitution."
return
}
if (!manifestVariablesFilesExist && manifestVariablesFilesExplicitlySpecified) {
// If the user explicitly specified a list of variables files, make sure they all exist.
// Otherwise throw an error so the user knows that he / she made a mistake.
error "[CFManifestSubstituteVariables] Could not find all given manifest variable substitution files. Make sure all files given as manifestVariablesFiles exist."
}
def result
ExecutionContext context = new ExecutionContext()
if (!manifestVariablesFilesExist && !manifestVariablesFilesExplicitlySpecified) {
// If no variables files exist (not even the default one) we check if at least we have a list of variables.
if (!manifestVariablesListSpecified) {
// If we have no variable values to replace references with, we skip substitution.
echo "[CFManifestSubstituteVariables] Could not find any default manifest variable substitution file at ${defaultManifestVariablesFileName}, and no manifest variables list was specified. Skipping variable substitution."
return
}
// If we have a list of variables specified, we can start replacing them...
result = substitute(manifestFilePath, [], manifestVariablesList, yamlUtils, context, debugHelper)
}
else {
// If we have at least one existing variable substitution file, we can start replacing variables...
result = substitute(manifestFilePath, manifestVariablesFiles, manifestVariablesList, yamlUtils, context, debugHelper)
}
if (!context.variablesReplaced) {
// If no variables have been replaced at all, we skip writing a file.
echo "[CFManifestSubstituteVariables] No variables were found or could be replaced in ${manifestFilePath}. Skipping variable substitution."
return
}
// writeYaml won't overwrite the file. You need to delete it first.
deleteFile(outputFilePath)
writeYaml file: outputFilePath, data: result
echo "[CFManifestSubstituteVariables] Replaced variables in ${manifestFilePath}."
echo "[CFManifestSubstituteVariables] Wrote output file (with variables replaced) at ${outputFilePath}."
}
}
/*
* Substitutes variables specified in files and as lists in a given manifest file.
* @param manifestFilePath - the path to the manifest file to replace variables in.
* @param manifestVariablesFiles - the paths to variables substitution files.
* @param manifestVariablesList - the list of variables data to replace variables with.
* @param yamlUtils - the `YamlUtils` used for variable substitution.
* @param context - an `ExecutionContext` to examine if any variables have been replaced and should be written.
* @param debugHelper - a debug output helper.
* @return an Object graph of Yaml data with variables substituted (if any were found and could be replaced).
*/
private Object substitute(String manifestFilePath, List<String> manifestVariablesFiles, List<Map<String, Object>> manifestVariablesList, YamlUtils yamlUtils, ExecutionContext context, DebugHelper debugHelper) {
Boolean noVariablesReplaced = true
def manifestData = loadManifestData(manifestFilePath, debugHelper)
// replace variables from list first.
List<Map<String>> reversedManifestVariablesList = manifestVariablesList.reverse() // to make sure last one wins.
def result = manifestData
for (Map<String, Object> manifestVariableData : reversedManifestVariablesList) {
def executionContext = new ExecutionContext()
result = yamlUtils.substituteVariables(result, manifestVariableData, executionContext)
noVariablesReplaced = noVariablesReplaced && !executionContext.variablesReplaced // remember if variables were replaced.
}
// replace remaining variables from files
List<String> reversedManifestVariablesFilesList = manifestVariablesFiles.reverse() // to make sure last one wins.
for (String manifestVariablesFilePath : reversedManifestVariablesFilesList) {
def manifestVariablesFileData = loadManifestVariableFileData(manifestVariablesFilePath, debugHelper)
def executionContext = new ExecutionContext()
result = yamlUtils.substituteVariables(result, manifestVariablesFileData, executionContext)
noVariablesReplaced = noVariablesReplaced && !executionContext.variablesReplaced // remember if variables were replaced.
}
context.variablesReplaced = !noVariablesReplaced
return result
}
/*
* Loads the contents of a manifest.yml file by parsing Yaml and returning the
* object graph. May return a `List<Object>` (in case more YAML segments are in the file)
* or a `Map<String, Object>` in case there is just one segment.
* @param manifestFilePath - the file path of the manifest to parse.
* @param debugHelper - a debug output helper.
* @return the parsed object graph.
*/
private Object loadManifestData(String manifestFilePath, DebugHelper debugHelper) {
try {
// may return a List<Object> (in case more YAML segments are in the file)
// or a Map<String, Object> in case there is just one segment.
def result = readYaml file: manifestFilePath
echo "[CFManifestSubstituteVariables] Loaded manifest at ${manifestFilePath}!"
return result
}
catch(Exception ex) {
debugHelper.debug("Exception: ${ex}")
echo "[CFManifestSubstituteVariables] Could not load manifest file at ${manifestFilePath}. Exception was: ${ex}"
throw ex
}
}
/*
* Loads the contents of a manifest variables file by parsing Yaml and returning the
* object graph. May return a `List<Object>` (in case more YAML segments are in the file)
* or a `Map<String, Object>` in case there is just one segment.
* @param variablesFilePath - the path to the variables file to parse.
* @param debugHelper - a debug output helper.
* @return the parsed object graph.
*/
private Object loadManifestVariableFileData(String variablesFilePath, DebugHelper debugHelper) {
try {
// may return a List<Object> (in case more YAML segments are in the file)
// or a Map<String, Object> in case there is just one segment.
def result = readYaml file: variablesFilePath
echo "[CFManifestSubstituteVariables] Loaded variables file at ${variablesFilePath}!"
return result
}
catch(Exception ex) {
debugHelper.debug("Exception: ${ex}")
echo "[CFManifestSubstituteVariables] Could not load manifest variables file at ${variablesFilePath}. Exception was: ${ex}"
throw ex
}
}
/*
* Checks if all file paths given in the list exist as files.
* @param manifestVariablesFiles - the list of file paths pointing to manifest variables files.
* @return `true`, if all given files exist, `false` otherwise.
*/
private boolean allManifestVariableFilesExist(List<String> manifestVariablesFiles) {
for (String filePath : manifestVariablesFiles) {
Boolean fileExists = fileExists filePath
if (!fileExists) {
echo "[CFManifestSubstituteVariables] Did not find manifest variable substitution file at ${filePath}."
return false
}
}
return true
}
/*
* Removes the given file, if it exists.
* @param filePath - the path to the file to remove.
*/
private void deleteFile(String filePath) {
Boolean fileExists = fileExists file: filePath
if(fileExists) {
Boolean failure = sh script: "rm '${filePath}'", returnStatus: true
if(!failure) {
echo "[CFManifestSubstituteVariables] Successfully deleted file '${filePath}'."
}
else {
error "[CFManifestSubstituteVariables] Could not delete file '${filePath}'. Check file permissions."
}
}
}