1
0
mirror of https://github.com/SAP/jenkins-library.git synced 2024-12-14 11:03:09 +02:00
Commit Graph

82 Commits

Author SHA1 Message Date
ffeldmann
b88ebdad6c
feat(splunk) Sending telemetry and logging information to Splunk (#2796)
* Adds inital splunk hook and logCollector

* Adds documentation of the Splunk hook

* Fixes markdown lint issues and removes comment from telemetry.go file

* Fixes markdown lint issues and adds missing generated file

* Markdown linting

* Changes documentation according to review, adds Splunk token automatically during init

* Adds error handling for marshalling hook config

* Markdown lint und correct Splunk token in httpclient

* Registeres Splunk token as secret and adjusts test cases

* Adds missing error handling and removes unnecessary comments

* Creates new function readPipelineEnvironment, adds tests

* Moves MonitoringData struct, edits defaults for json fields

* Adds gitRepository and gitOwner to telemetry information

* Simplifies readCommonPipelineEnvironment function, adds more descriptive errors, adds automated adding of Splunk prefix token

* Adjusts error handling

* Cleaner error logging
2021-05-17 12:14:04 +02:00
lndrschlz
4ca9186f39
fix(uiveri5ExecuteTests): add 'tests' stash to step yaml (#2641)
* add tests stash to uiveri5 step

* add stash to uiveri5

* extend step generator with input resources

* add step generator test
2021-03-01 13:03:42 +01:00
Fabian Reh
d8d4f83fb4
Unit test enhancement (#2471)
* Fixes parallelization in generated code

* Fixes test_code_generated.golden

* generate go code

Signed-off-by: Fabian Reh <fabian.reh@sap.com>

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-12-22 09:30:54 +01:00
Marcus Holl
f3f221ee80
generator helper: support for map[string]interface{} (#2232)
Up to now the code generator is not able to handle the type
map[string]interface{} which is important for nested
configurations.

With that change we support such nested configuration.

Fo now parameters with a map type are not supported via
command line parameters. Those parameters are simply
ommitted. But with this change is it possible to read
such nested structures from the pipeline configuration
(.pipeline/config.yml).

As a next step we can discuss if we would like to support
such values also via command line parameters. One possible
approach could be

```
./piper <command> -myParam key1=val1 --myParam key2=val2
```

which gets finally collected inside our map:

```
map["key1"] = "val1"
map["key2"] = "val2"
```

This is of course hard to do for deeper nestings. In that case
providing a pointer to a file might be more suitable.

In that context we need to consider how to
  - declare the default values for map like parameters in our
    metadata files.
  - deal with the different types we have for the parameter
    itself wrt the yaml like config on the one hand and on the
    level of the command line parameters on the other hand. Maybe for
    that we have to extend the metadata format (e.g. describe an
    alternate type receiving the values from the command line, like
    []string. With that approach values for simple nested (... not deep
    nested) params can be provided like described above, it would be
    possible to represent these parameters for the command line parser
    as string slice entries like "[]string{key1=val1, key2=val2". These
    parameters needs in this case transformed "by us" into the map we
    use further down the road.

In case we agree in principle on an approach as outlined here we should
adjust the golden files reflecting this use case.
2020-11-23 07:55:20 +01:00
lndrschlz
b9bab27833
feat: expose complete step metadata through generated function (#2329)
* exposing step metadata through generator

* add metadata_generated.go

* fix step go test generation

* metadata fields added to generated files

* added generated files

* removed image placeholder from fortify step

* refactored step meta generation

* go generate

* fixed metadata generator and tests

* added output resource fields/tags to metadata generator

* fix string in metadata_generated

* go generate

* fixed generator

* go generate

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-11-20 08:13:59 +01:00
Marcus Holl
073a2a8599
generator: Don't swallow output from go fmt (#2222)
* generator: Don't swallow output from go fmt

* fix formatting

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-11-02 10:31:00 +01:00
Oliver Nocon
26cfbf7357
fix(whitesourceExecuteScan) properly handle output resources (#2266)
* fix(whitesourceExecuteScan) properly handle output resources

* fix merge issues

* add required aliases

* update generation

* fix reading custom and container environment parameters from cpe

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
2020-11-02 08:51:58 +01:00
Stephan Aßmus
582419e2f5
Up for discussion: Embed best practices (?) in generated steps (#1913)
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-10-29 17:29:07 +01:00
Kevin Stiehl
24aafb0b69
add vaultSecretFileReferences (#2203)
* add vaultSecretFileReferences

* fix test

* fix test

* go generate

* remove code duplication

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-10-26 14:20:04 +01:00
Christopher Fenner
f3a4342261
feat(influx): handle data types of non-string fields (#2132)
* add type to sonar field

* respect type of influx fields

* update generated code

* switch type

* copy changes from #1885

* log JSON data

* read simple values from json

* Update InfluxData.groovy

* Revert "Update InfluxData.groovy"

This reverts commit c8cfdf381f.

* Revert "read simple values from json"

This reverts commit 94b69866d2.

* Revert "copy changes from #1885"

This reverts commit 2471b4475e.

* update TODO
2020-10-13 16:37:48 +02:00
Christopher Fenner
be90876b7c
feat(output): handle non-string output values (#2113)
* handle non-string values as JSON

* change value type to interface in resources

* regenerate code

* add test cases

* handle reading of json files

* write json data to json files

* fix assignment

* use GetResourceParameter

* add test case

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
2020-10-05 15:33:28 +02:00
Oliver Nocon
19c1732826
Telemetry: report error category (#2085) 2020-09-29 13:49:40 +02:00
Christopher Fenner
b219fb6514
fix(typo): found by misspell (#2064)
* fix typos in step yamls

* fix typos in go files

* regenerate step code

* fix typos in md files

* fix typos in groovy files

* fix further typos
2020-09-24 07:41:06 +02:00
Christopher Fenner
54ff93233e
refactor(docs): separate step and docs generator (#2033)
* remove docs generator code from step-generator

* add docs generator to dedicated package

* add test cases

* add entry point for docs generation

* make output more readable

* remove dead code
2020-09-23 13:55:17 +02:00
Kevin Stiehl
d589038206
Vault AppRole login (#1971)
* added interpolation package in config

* vault allow paths to hold config references

* allow referencing properties in vaultPaths

* fix small typo

* add approleAuth

* register resolved secrets to logger

* generate steps

* clean up

* add integration test

* add vault to context filter

* reduce Cognitive Complexity & added tests

* Update pkg/config/stepmeta_test.go

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>

* go generate

* go generate after merge

* rename VaultAppRole* to VaultRole*

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-09-16 14:50:09 +02:00
Oliver Feldmann
41abec6b10
Fix generator typo (#2008)
* Fix typo in generator code

* Fix typo in generated code
2020-09-10 14:37:46 +02:00
Christopher Fenner
666923e651
docs: change secret docs (#1984)
* change secret docs

* link to credentials in resource reference

* cleanup

* correct type

* add configuration scope

* correct test case
2020-09-10 10:33:59 +02:00
Oliver Nocon
e09b05b22e
DocGenerator: fix links to secrets (#1825)
* DocGenerator: fix links to secrets
* Update docuHelper.go
2020-07-22 10:33:36 +02:00
Oliver Nocon
220748d150
Step generation: allow usage of go generate (#1793)
* Step generation: allow usage of go generate

* Update DEVELOPMENT.md

* move generate command to main.go (#1809)

* Update main.go

* Update DEVELOPMENT.md

* Update piper.go

* align path usage

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-07-17 08:06:11 +02:00
Oliver Nocon
c622b3dfac
Documentation: fix parameter links & some spelling (#1807)
* Documentation: fix parameter links & some spelling
* Fix new line character
* update checkboxes
2020-07-16 11:34:02 +02:00
Oliver Nocon
43f51ba90a
Revamp documentation generation for golang steps (#1781)
* Revamp documentation generation for golang steps

* Add and update tests

* Add tests, add step outputs

* Add tests, add step outputs

* Update rendering

* fix merge conflict

* update generated files

* Update pkg/config/stepmeta.go

* Update pkg/config/stepmeta.go

* Update pkg/generator/helper/docuHelper.go

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>

* Include PR feedback

* update tests

* Fix Jenkins dependecy indication

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-07-16 09:10:15 +02:00
Oliver Nocon
eafe383d54
Add error category parsing to cmd execution (#1703)
* Add error category parsing to cmd execution

It is now possible to define `ErrorCategoryMapping` as a `map[string][]string` on a `Command`.
The format contains the category as key which has a list of error patterns assigned.
Example:

```
cmd := Command{
  ErrorCategoryMapping: map[string][]string
    "build": {"build failed"},
    "compliance": {"vulnerabilities found", "outdated components found"},
    "test": {"some tests failed"},
  },
}
```

Setting this map triggers console log parsing when executing a command.
If a match is found the error category is stored and
it will automatically be added to the `errorDetails.json`.

* clean up go.mod

* fix test

* fix test

* Update DEVELOPMENT.md

* fix tests

* address long console content without line breaks

* scan condition update

* fix test

* add missing comment for exported function

* Update pkg/command/command.go

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-06-24 10:04:05 +02:00
Christopher Fenner
c42553593e
fix: remove unused variables from generated step coding (#1698)
* remove unused parameters from generated code

* update steps

* update steps

* correct golden files

Co-authored-by: Stephan Aßmus <stephan.assmus@sap.com>
2020-06-23 18:05:21 +02:00
Maximilian Lenkeit
10e5bf98f8
artifactPrepareVersion: remember original artifact version (#1600)
* Update resources.go

* expose originalArtifactVersion parameter

* re-generate steps

* adjust golden files for generator changes
2020-05-28 10:05:22 +02:00
Sven Merk
af2a01c064
Fortify implementation in golang (#1428) 2020-05-25 19:48:59 +02:00
Oliver Nocon
7f7d0bce89
Piper steps: Issue SUCCESS message (#1478)
* Piper steps: Issue SUCCESS message

Log `SUCCESS` message in all steps by default if no error occured

* Update additional generated steps

* Update generated step
2020-05-18 21:52:54 +02:00
Oliver Nocon
37a6d456fe
Fix generator for custom steps (#1522) 2020-05-08 10:41:01 +02:00
Stephan Aßmus
082b249cc0
Fix logrus buffer issue (#1511) 2020-05-06 13:35:40 +02:00
lndrschlz
39e30ff1b4
Added hook configuration; added sentry hook and tests (#1497)
* Added hook configuration; added sentry hook and tests
* steps generated by step framework
* added comments and renamed correlationID
* formatted comment on NewSentryHook method
* fix generator step tests
2020-05-05 08:36:24 +02:00
Christopher Fenner
6eb8dc8191
docs: add possible value for bool parameters (#1486) 2020-04-30 06:39:48 +02:00
Christopher Fenner
3a4d4b2de7
docs: add general config parameters to GO docs (#1471)
* extract StepConfiguration generation to separate file

* add general options to parameter and config table

* fix test case

* remove telemetry flag from docs

* Revert "extract StepConfiguration generation to separate file"

This reverts commit df80dab8cd.
2020-04-29 14:40:11 +02:00
Oliver Nocon
43947e6ef5
Pass golang error details to Jenkins pipeline (#1443)
Do not exit with os.Exit(1) but using log.Entry().Fatal() instead
* Golang: forward error details
* extend groovy wrapper to provide proper error message
* create closure for error handling
2020-04-28 07:42:02 +02:00
Christopher Fenner
8455306a5e
docs: add code blocks to parameters table (#1460)
* add code blocks

* set parameters name in code block

* change test cases

* set parameters name in code block

* add step configuratio caption

* remove details caption

* add code block for docker settings

* add code block for stash content

* add code block for defaults

* add code block for sidecar image & name

* correct test cases

* add code block for docker

* correct test cases
2020-04-27 12:16:41 +02:00
Christopher Fenner
1660f4df48
docs: remove dangling whitespaces (#1459)
* omit <nil> in parameter table

* remove additional whitespaces

* add newline

* remove whitespaces

* adjust test cases

* fix whitespaces
2020-04-24 21:48:28 +02:00
Christopher Fenner
96439972e2
docs: show possible values in step documentation (#1453)
* add possibleValues to model

* include possible values into docs
2020-04-24 14:13:02 +02:00
Christopher Fenner
2100632b54
docs: omit <nil> in parameter table (#1450)
* omit <nil> in parameter table

* adjust test case
2020-04-24 11:21:33 +02:00
Oliver Nocon
75a10742d3
Step generator: correct error handling (#1429)
Do not exit with os.Exit(1) but using log.Entry().Fatal() instead
2020-04-20 16:52:57 +02:00
Daniel Kurzynski
ce4be51362
Mask secrets (#1382) 2020-04-16 14:37:45 +02:00
Christopher Fenner
60b7d3a8df
refactor(sonar): implement Sonar step in GO (#1042)
* refactor(sonar): add initial metadata file for GO-based sonar step

* refactor(sonar): add initial generated files for GO-based sonar step

* refactor(sonar): add initial step file for GO-based sonar step

* refactor(sonar): add sonar cmd to executable

* refactor(sonar): cleanup metadata file

* fix(go): use capital TLS in golang names

* add download of Sonar scanner cli

* move download & unzip methods to FileUtils

* download certificate files

* use http.Downloader

* update generated code

* add slice prefix function

* update sonar step

* add test cases

* update docs

* use node:lts image

* use latest sonar-scanner

* add test cases

* use latest sonar-scanner

* fix test case

* change things

* return errors on functions

* limit changeID visibility

* use lowercase scm provider

* add test case for legacy PRs

* add test case

* update docs

* use local temp dir

* extend parameter visibility to GENERAL

* use SONAR_AUTH_TOKEN

* remove comments

* update generated code

* Apply suggestions from code review

* change to private fields

* use setter for sonar options & environment

* remove obsolete return statement

* add comment about Downloader mock

* add test case for custom options

* handle custom options

* refactor slice functions

* fix go fmt

* fix generated step code

* update docs
2020-03-23 10:38:31 +01:00
Oliver Nocon
8fbeddb26c
Golang step metadata: Config aliases for steps (#1293)
* Golang step metadata: Config aliases for steps

This will ease following scenarios:
* config migration due to step name changes
* re-use of more general config, e.g. `mavenExecute` in `mavenBuild`

* fix CodeClimate finding

* Fix panic if original stage config does not exist yet
2020-03-19 17:24:35 +01:00
Florian Geckeler
48fa0a7521
Add 'Code generated by' header to stepGoTemplate 2020-02-21 12:56:09 +01:00
Oliver Nocon
a5d4c64bf4
Correct step generator (#1190) 2020-02-18 08:31:54 +01:00
Oliver Nocon
ec4fb1b186
Update generator (#1188)
Provide better starting point for new steps
2020-02-17 13:30:30 +01:00
Oliver Nocon
9c1bd04752
Streamline step generation (#1142)
* Streamline step generation
* Include PR feedback, update DEVELOPMENT.md

Co-authored-by: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2020-02-04 10:46:43 +01:00
Christopher Fenner
768c6aa399
fix: generator for custom steps (#1139) 2020-01-31 15:39:45 +01:00
Oliver Nocon
b542d15998
Enhance telemetry reporting (#1133)
* Enhance telemetry reporting
* Use central telemetry data object
* Add duration

Co-Authored-By: Christopher Fenner <26137398+CCFenner@users.noreply.github.com>
2020-01-31 14:06:08 +01:00
Christopher Fenner
aa3fb8adb4
feat(go): add telemetry reporting (#1100)
* Add telemetry support

* First round telemetry

* Add telemetry flag

* fix: move files to avoid import cycles

* add noTelemetry as global config option

* Respect telemetry configuration for reporting

* add site id, swa endpoint

* correct logger initialization

* add http logic

* rename init method

* rename consts & types

* convert struct to payload

* convert data to payload string

* move activation flag out of data structure

* extract types to own file

* build query using net/url

* correct field mapping

* extract notify coding to own file

* cleanup parameter mapping

* preare base data

* fix codeclimate issue

* correct test case

* fill values from env

* test all fields

* untrack notify.go

* ignore empty custom values

* cleanup data.go

* add test cases

* cleanup

* add usage reporting to karma step

* add usage reporting to step generator

* externalise siteID

* correct custom field names

* test env handling

* simplify method signature

* revert parameter negation

* correct import

* adjust golden file

* inclease log level

* ignore test case

* Revert "inclease log level"

This reverts commit 70cae0e029.

* add test case for envvars

* remove duplicate reporting

* remove duplicate reporting

* correct format

* regenerate checkmarx file

* add log message on deactivation

* rename function

* add comments to understand SWA mapping

Co-authored-by: Oliver Nocon <33484802+OliverNocon@users.noreply.github.com>
2020-01-29 13:17:54 +01:00
Oliver Nocon
73ab887f25
Add kubernetesDeploy step (#1073)
* Add kubernetesDeploy step

Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2020-01-24 14:30:27 +01:00
Oliver Nocon
a46b57e6b4
Persisted pipeline environment for golang library (#1091)
* Use commonPipelineEnvironment in go binary

* Update groovy part incl. tests

* Rework structure and naming

* Support influx resources in steps

* Update tests and some cleanups

* Add correct defer handling

* Address PR feedback

* Fix test

* Update resources.go

Co-authored-by: Sven Merk <33895725+nevskrem@users.noreply.github.com>
2020-01-15 12:16:25 +01:00
Oliver Nocon
4f57738888
Detect: Add golang implementation (#1049)
Add detect golang implementation
2019-12-13 09:55:45 +00:00