* makes containerImage not mandatory
* Adds kubectl container
* Adds log statement to debug
* adds general to container image
* removes GENERAL again
Removes condition from Kubectl container
* removes workDir
* marks logs as debug
* adds workingdir again
* Adds author to commits
* Adds commit time now
* remove deprecated and reorder
* adds deprecated again to containerRegistryUrl
Adds GENERAL scope to containerImage
* updates generated file
* Renames containerImageNameTag
* adds else case
* adds debug log
* code cleanup
* adds debug log
* revert
* adds debug logs
* revert
* makes root path not hidden
* revert
* Read container properties
* Removes debug message
* Removes debug message
* Removes general scope again
* Fixes unit test
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* add vaultSecretFileReferences
* fix test
* fix test
* go generate
* remove code duplication
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* Adds piper step to update deployment configuration in external git repository.
https://github.wdf.sap.corp/ContinuousDelivery/piper-ita/issues/21
* Adds handling of branchName as an optional parameter
* Update resources/metadata/gitopsUpdateDeployment.yaml
Feedback about description
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Adapt to interface guide
* Refactors to GitopsExecRunner
* Refactors to GitopsExecRunner in test
* Removes unnecessary mocked methods
* Adds tests for git utils
* Adds new step to CommonStepsTest.groovy
* Updates description from yaml
* Restricts visibility of methods and interfaces
Adds comments where necessary
* Updates comments
* Fixes URL name
* updates description
* updates generated file
* Fixes compile issue in CommonStepsTest.groovy
* Updates long description
* Updates test to run green on all kind of OS
* Removes global variables from tests
* Default branch: master
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Typo in Hierarchy
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Refactors test to allow parallel execution
* Renames utility variable in gitopsUpdateDeployment.go
* Renames error variables in gitopsUpdateDeployment.go
* simplified parameters for kubectl
* Refactors util classes to use parameters rather than global variables
* makes username and password mandatory
* remove unnecessary mandatory flag
* remove new methods from mock that are not necessary
* replaces with EqualError
* replaces with NoError
* update generated file
* refactor tests
* refactor tests
* make tests parallel executable
* parallel execution of tests
* Refactors interfaces to stop exposing interfaces
* Feedback from PR
* Simplifies failing mocks
* Renames variables and interfaces
* Fixes error messages
* shorten variable names
* Renames unused parameters in tests
* Cleanup nil parameters
* Typo
* Wrap errors and remove unnecessary logs
* Remove containername and filePath from GENERAL scope
* correct generated file
* corrects expected error messages
Co-authored-by: OliverNocon <oliver.nocon@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* remove docs generator code from step-generator
* add docs generator to dedicated package
* add test cases
* add entry point for docs generation
* make output more readable
* read additional defaults
* add custo defaults parameters
* remove commented code
* adjust custom default parameter in workflow
* remove conflict leftovers
* handle custom default values
* remove comment
* extract code to function
* extract metadata reading to function
* do not print empty strings in favor of PIPER_* env vars
* extract new code to own metadata file
* only reset default on booleans
* remove obsolete test case
* kanikoExecute: improve user experience
* ensure proper tags
* update permissions
in case a container runs with a different user
we need to make sure that the orchestrator user
can work on the file
* update permissions
* ensure availablility of directories on Jenkins
* (fix) clean up tmp dir in test
* add resilience for incorrect step yaml
* incorporate PR feedback
* add type to sonar field
* respect type of influx fields
* update generated code
* switch type
* copy changes from #1885
* log JSON data
* read simple values from json
* Update InfluxData.groovy
* Revert "Update InfluxData.groovy"
This reverts commit c8cfdf381f.
* Revert "read simple values from json"
This reverts commit 94b69866d2.
* Revert "copy changes from #1885"
This reverts commit 2471b4475e.
* update TODO
* handle non-string values as JSON
* change value type to interface in resources
* regenerate code
* add test cases
* handle reading of json files
* write json data to json files
* fix assignment
* use GetResourceParameter
* add test case
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* add code block type
* add parameters to hand in library and binary name
* use library and binary name parameters
* add test cases
* use yaml file to distintuish custom from regular steps
* add test case
Vars file handling centralized
We have the same coding for handling varsf-files and vars. With that change we shift to having one common coding for that
* remove docs generator code from step-generator
* add docs generator to dedicated package
* add test cases
* add entry point for docs generation
* make output more readable
* remove dead code
* fix redundant type issues
* cleanup
* extract report function for protecode package
* use speaking status constant for API results
* remove unconsidered return value
* correct switch statement
* handle severe vulnerabilities
* Apply suggestions from code review
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* correct test name
* return errors from WriteReport function
* expose ReportData struct
* set Error Category
* refactor constant visibility
* change type name
* describe type
* change type name
* fail after report generation
* do not fail on report write errors
* add error as return value
* fix typo
* use require statements
* assert major vulnerabilities
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Fix
* Adapt errors
* Consider unexpected JSON
* defer closing the response body
* Add comments to explain function
* Improve assert statements semantically
* Change comment format due to CodeClimate
* Extract sub function
The SAP NPM registry has been migrated to the default public registry,
thus the separate configuration with the sapNpmRegistry is not required
anymore.
All packages from npm.sap.com have been migrated to npmjs.org
and in the future SAP packages will only be available from the default
public registry.
Currently, the mtaBuild step installs the wrong artifact in a spring project making use of the "repackage" feature. This PR fixes that by checking if an ".original" jar file exists and using that instead.
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
* Add abapEnvironmentCheckoutBranch step setup
* Change description of abapEnvCheckoutBranch step
* Add http client code
* Disable code due to missing interace
* Add coding for use of abaputils
* Adapt checkout branch step
* Adapt URL for checkout_branch function import
* Fixes
* Fix for missing mapping of CfSpace
* Add working code for a Branch Checkout
* Adapt changes of abautils pkg
* Add test for polling
* Minor fix
* Fix yaml spacing
* Add longdescription to yaml
* Refactor abaputil methods
* Refactoring
* Refactoring
* Minor fix
* Minor fixeds
* Adapt to new abaputils.AUtilsMock
* Delete obsolete initial checks for params
* Fix manageGitRepoUtils_test.go
* Adjust pollEntity tests
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
* Add abaputils pkg and go files
* Add ReadServiceKeyAbapEnvironment function
* Fixes
* Add structs for SC, Pull and Branch
* Minor Improvements
* Adapt unit tests to new abaputils pkg
* Fixes
* Add adapted tests
* Fixes
* Fix cloudfoundry test
* Add check for host prefix (HTTPS)
* Fix tests + cleanup
* Fixes
* Fixes
* Fix
* Add mock for abaputils pkg unit tests
* Adapt abaputils comments
* Add unit test for missing params case
* Fix for missing mapping of CfSpace
* Fix host schema
* Remove LogoutOption param of unit tests and steps
* Fix unit test
* Fix unit test CF ReadServiceKey
Co-authored-by: Daniel Mieg <56156797+DanielMieg@users.noreply.github.com>
This change adds a buildDescriptorExcludeList parameter to
npmExecuteScripts, to enable the exclusion of certain directories when
executing npm scripts. Previously, npmExecuteScripts could only execute
scripts in all packages.
Now it is possible to provide paths or patterns as elements of the
buildDescriptorExcludeList to exclude packages when executing npm scripts.
* Dont work upon a global command.Command instance inside cloudfoundry package
o Up to now we work on a private and shared instance of command.Command inside
the cloudfounrdy package. We need to be able either configure this instance
(environment variables) according to the use case. One option is to hand over
an already configured instance which is used elsewhere. This is what we do with
this commit.
o With this commit we remove the instance which is shared within the cloudfounrdy
package and to provide an instance with a receiver which gets handed over to the
functions. Hence we are thread save: parallel invoctation of e.g. Login will not
affect each other.
o Up to now we work on a private and shared instance of command.Command inside
the cloudfounrdy package. We need to be able either configure this instance
(environment variables) according to the use case. One option is to hand over
an already configured instance which is used elsewhere. This is what we do with
this commit.
o With this commit we remove the instance which is shared within the cloudfounrdy
package and to provide an instance with a receiver which gets handed over to the
functions. Hence we are thread save: parallel invoctation of e.g. Login will not
affect each other.
* Added Vault package
* added support for logical path lookups instead of api paths
* added integration tests
* add integration tests and mock tests
* Replace mock with mockery generated one
* update tests to use mockery
* create mocks sub package
- mock is used now for cf api commands
- tests for cf api and cfk login are checking which
commands are really executed
- some minor simplifications wrt asserts
* [refactoring] move the shell/command related interfaces into pkg/command
otherwise we are not able to use the corresponding mocks for the items contained in pkg since
these interfaces are not visible from the pkg folder
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Whitesource MVP for Gradle, Golang, and NPM/Yarn
* Refactoring
* Refactor and cleanup, better error checking
* publish stepResults, use pkg/versioning, bubble up errors, add gomod versioning support
* Run gofmt and cleanup comments
* Resolve PR comments
* Update resources/metadata/whitesource.yaml
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Only determine project coordinates if they are missing
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Gradle versioning artifact
* fix gradle artifact version regexp and refactor
* Fix token extraction from output buffer
* Fix some issues with pip and jsonfile versioning logic
* Remove useless spacing
* Remove unnecessary test file and fix naming style for JSONDescriptor
* Automatically download wss-unified-agent if file does not exist
* adds downloadVulnerabilityReport, checkSecurityViolations, minor refactoring
* adds config.ReportDirectoryName, improves readability
* Version-wide reporting for vulnerabilities and list of libraries.
* Refactor and improve build accuracy
* fix sed command
* Add includes file pattern config option
* Adds --exclude command line flag
* run go mod tidy and regenerate step framework
* Fix unit tests
* revert changes
* poll project status before downloading reports
* merge with master
* go mod tidy, go fmt, and fix whitesource unit test
* sync go.mod
* sync go.mod again
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
* Allow retrieving exit code from command execution
This will be helpful to derive error categories in case
an executable provides context-specific error codes.
* make sure that we always have a non 0 exit code for errors
* Add capabilities for checks if a file has been written
With the current file system mock we cannot assert if
a file has been written. E.g. we cannot distiguish between
files added to the virtual file system before the test and files
explicitly written. In contrast to that we can check for deleted
files.
With the change here we get a func HasWritteFile(name).
[Q] Wouln't it be possible to check based on the file content
if a file has been written (the new file should have another
content as the file registered before).
[A] We should not assert some file content here since the
produced file content can be created by another "class" which
is unit tested somewhere else. With that approach we would test
the producer here again.
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
* Add error category parsing to cmd execution
It is now possible to define `ErrorCategoryMapping` as a `map[string][]string` on a `Command`.
The format contains the category as key which has a list of error patterns assigned.
Example:
```
cmd := Command{
ErrorCategoryMapping: map[string][]string
"build": {"build failed"},
"compliance": {"vulnerabilities found", "outdated components found"},
"test": {"some tests failed"},
},
}
```
Setting this map triggers console log parsing when executing a command.
If a match is found the error category is stored and
it will automatically be added to the `errorDetails.json`.
* clean up go.mod
* fix test
* fix test
* Update DEVELOPMENT.md
* fix tests
* address long console content without line breaks
* scan condition update
* fix test
* add missing comment for exported function
* Update pkg/command/command.go
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
* Add possibility to add category to failures
It is now possible to set the error category within the flow.
When exiting the program the error category can be used.
There is a convenience function available for exiting with a previously set category,
for example
```
log.SetErrorCategory(log.ErrorCompliance)
...
log.FatalError(err, "configuration error")
```
* extend test
* go mod tidy
* add missing comment
* update information about error categories
* Update DEVELOPMENT.md
This change refactors the npm pkg and npmExecuteScripts implementations
to be reusable for future steps, e.g., npmExecuteLint.
In addition, it fixes few small bugs related to unit test execution on
Windows and the fileUtils mocking implementation.
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
There are use cases where we need to run `GetStepConfig()` multiple times.
In such cases it is more efficient to load the files once and then resolve the
respective step configuration.
Extend mta build step to install maven artefacts after build to allow re-using them in later stages (additional unit or integration tests which might not be running as part of the "build" life-cycle).
* also reduce code duplication in token fetching
* concatenate classpaths from multi-maven projects
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* Include error in the log.
In case the default text formatter is used, the error
will be contained in the log, too.
* Avoid stupid nil token printed in error message
* Add parameter "--ignoreCustomDefaults"
* Pass to piper customDefaults from config also via --defaultConfig
... and add "--ignoreCustomDefaults".
* Log output when ignoring customDefaults
Co-authored-by: Daniel Kurzynski <daniel.kurzynski@sap.com>
* artifactPrepareVersion: additional options
it is now possible to use
* a Unix timestamp number
* a short commit hash
* Add dedicated Docker versioning scheme
Avoid maven error `Unknown lifecycle phase \"-\"` when the value of a define contains `-`.
Don't split and trim maven arguments. Expect they come in as a list, keep them as list.
This is a breaking change compared to the old Groovy implementation which relied on using a shell for calling maven.
As an example, consider this diff:
```diff
- goals: 'org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate',
- defines: "-Dexpression=$pomPathExpression -DforceStdout -q",
+ goals: ['org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate'],
+ defines: ["-Dexpression=$pomPathExpression", "-DforceStdout", "-q"],
```
* extract StepConfiguration generation to separate file
* add general options to parameter and config table
* fix test case
* remove telemetry flag from docs
* Revert "extract StepConfiguration generation to separate file"
This reverts commit df80dab8cd.
The TransportTimeout value is now applied to the "response header
timeout" and "expect continue timeout" as is. Previously there
was a hard limit of 10 seconds and 1 second respectively (originating
from the article I based the previous PR on). While this doesn't
allow for fine-grained control, it is reasonable to apply the meaning
of "transport timeout" to any state or phase of a HTTP connection.
This change should solve the needs of some Piper clients to configure
very long response header timeouts of 5 minutes and above.
Do not exit with os.Exit(1) but using log.Entry().Fatal() instead
* Golang: forward error details
* extend groovy wrapper to provide proper error message
* create closure for error handling
* add code blocks
* set parameters name in code block
* change test cases
* set parameters name in code block
* add step configuratio caption
* remove details caption
* add code block for docker settings
* add code block for stash content
* add code block for defaults
* add code block for sidecar image & name
* correct test cases
* add code block for docker
* correct test cases
