1
0
mirror of https://github.com/offen/docker-volume-backup.git synced 2025-05-24 14:01:19 +02:00

83 Commits

Author SHA1 Message Date
Frederik Ring
63961cd826
Pass file location to lifecycle commands (#173)
* Add test case for extending image and calling through to rsync

* Keep backup file location env var

* Add documentation

* Work against untared content in test
2022-12-30 16:07:34 +01:00
Frederik Ring
9534cde7d9
Allow use of a custom ca cert when working against S3 storages (#170) 2022-12-22 14:37:51 +01:00
Frederik Ring
5c7856feb3 Consider failed casting to error response, use established minio bootstrap in tests 2022-10-13 19:40:41 +02:00
pixxon
b5cc1262e2
add aws secret handling (#161)
* add aws secret handling

* make it look go-ish

* fix tests

* whitespace

* sleep a bit
2022-10-12 19:14:57 +02:00
Frederik Ring
00c83dfac7 Fix more error strings 2022-09-15 10:49:45 +02:00
Frederik Ring
a6ec128cab
Run copying and pruning against multiple storages in parallel (#152) 2022-09-01 14:38:04 +02:00
Frederik Ring
b60c747448 Fix WebDAV spelling, remove some inconsistencies (#143)
* Simplify logging, fix WebDAV spelling

* Define options types per package

* Move util functions that are not used cross package

* Add per file license headers

* Rename config type
2022-08-18 12:37:45 +02:00
MaxJa4
279844ccfb Added abstract helper interface for all storage backends (#135)
* Added abstract helper interface and implemented it for all storage backends

* Moved storage client initializations also to helper classes

* Fixed ssh init issue

* Moved script parameter to helper struct to simplify script init.

* Created sub modules. Enhanced abstract implementation.

* Fixed config issue

* Fixed declaration issues. Added config to interface.

* Added StorageProviders to unify all backends.

* Cleanup, optimizations, comments.

* Applied discussed changes. See description.

Moved modules to internal packages.
Replaced StoragePool with slice.
Moved conditional for init of storage backends back to script.

* Fix docker build issue

* Fixed accidentally removed local copy condition.

* Delete .gitignore

* Renaming/changes according to review

Renamed Init functions and interface.
Replaced config object with specific config values.
Init func returns interface instead of struct.
Removed custom import names where possible.

* Fixed auto-complete error.

* Combined copy instructions into one layer.

* Added logging func for storages.

* Introduced logging func for errors too.

* Missed an error message

* Moved config back to main. Optimized prune stats handling.

* Move stats back to main package

* Code doc stuff

* Apply changes from #136

* Replace name field with function.

* Changed receiver names from stg to b.

* Renamed LogFuncDef to Log

* Removed redundant package name.

* Renamed storagePool to storages.

* Simplified creation of new storage backend.

* Added initialization for storage stats map.

* Invert .dockerignore patterns.

* Fix package typo
2022-08-18 12:37:45 +02:00
MaxJa4
2b7f0c52c0
Print more error info for minio (#136)
* Print more error info for minio

* Unpacked error info
2022-08-15 09:25:32 +02:00
Frederik Ring
b441cf3e2b
Fine grained labels (#115)
* Refactor label command mechanism to be more flexible

* Run all steps wrapped in labeled commands

* Rename methods to be in line with lifecycle

* Deprecate exec-pre and exec-post labels

* Add documentation

* Use type alias for lifecycle phases

* Fix bad imports

* Fix command lookup for deprecated labels

* Use more generic naming for lifecycle phase

* Fail on erroneous post command

* Update documentation
2022-07-10 10:36:56 +02:00
Erwan LE PRADO
3661a4b49b
feat: Add storage class header (#119)
* feat: Add storage class header

* doc: change the readme

* chore: Remove the unnecessary default  value
2022-07-06 13:18:12 +02:00
Frederik Ring
342ae5910e
Add env template helper (#121) 2022-07-06 13:16:32 +02:00
Frederik Ring
1892d56ff6
Change default value for SSH identity file (#108)
* Change default value for SSH identity file

* Force remove write protected file in tests
2022-06-17 11:28:29 +02:00
İbrahim Akyel
0b205fe6dc
SSH Backup Storage Support (#107)
* SSH Client implemented

* Private key auth implemented
Code refactoring

* Refactoring

* Passphrase renamed to IdentityPassphrase
Default private key location changed to .ssh/id
2022-06-17 11:06:15 +02:00
Frederik Ring
b52b271bac
Allow for the exclusion of files from backups (#100)
* Hoist walking of files so it can be used for features other than archive creation

* Add option to ignore files from backup using glob patterns

* Use Regexp instead of glob for exclusion

* Ignore artifacts

* Add teardown to test

* Allow single Re for filtering only

* Add documentation

* Use MatchString on re, add bad input to message in case of error
2022-05-08 11:20:38 +02:00
Frederik Ring
94a1edc4ad
Allow disabling of certificate verification for WebDAV (#98) 2022-04-20 14:16:59 +02:00
Frederik Ring
1b1fc4856c
List objects recursively when selecting candidates from S3 (#92) 2022-04-15 11:05:52 +02:00
Frederik Ring
e81c34b8fc
Consider S3 Path when selecting candidates for pruning (#91) 2022-04-13 17:09:37 +02:00
Frederik Ring
51af8c3c77
Deprecate BACKUP_FROM_SNAPSHOT (#81) 2022-03-25 18:28:58 +01:00
Frederik Ring
da8c63f755
Support identical cron schedule (#87)
* Retry on lock being unavailable

* Refactor locking to return plain error

* Collect LockedTime in stats

* Add test case

* Add documentation for LOCK_TIMEOUT

* Log in case lock needs to be awaited

* Release resources created for awaiting lock
2022-03-25 18:26:34 +01:00
Frederik Ring
b8f71b04a1
Use errgroup for running commands in parallel (#83) 2022-03-10 11:09:39 +01:00
Frederik Ring
5f3832d621
Consider prefix rules when pruning WebDAV storages (#79) 2022-03-05 13:33:15 +01:00
Frederik Ring
ae50a3ac4f Add attribution to code taken from moby repository 2022-03-04 16:40:34 +01:00
Frederik Ring
9155b4d130 Add missing print directive, fix go.mod 2022-02-23 10:12:57 +01:00
Frederik Ring
0504a92a1f
Add option to run pre/post commands for any container (#73)
* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
2022-02-22 07:53:33 +01:00
Frederik Ring
3ded77448c
Do not skip directories when creating tar archive (#72)
* Update targz library to include potential ownership fix

* Move archive logic to main repo

* Remove assertions for debugging

* Use relative path in assertion

* Strip local part from archive location

* Log when extracting in tests

* Fix trimming of prfix

* Add license info to archive.go file

* Undo change in test assertion

* Add test checking for preserved file ownership

* use same postgres version in tests

* Wrap errors when archiving, handle deletion at script layer
2022-02-22 07:49:24 +01:00
Frederik Ring
58b42b9036
Supporting proxied Docker APIs through DOCKER_HOST (#70) 2022-02-18 09:08:21 +01:00
Mauro Molin
30265c14ba
Fixed TookTime (#66) 2022-02-14 17:32:05 +01:00
Frederik Ring
a57e93d01e
Split source into multiple files, deduplicate pruning logic, do not parse templates when notifications are not used (#63)
* Split code into multiple files

* Deduplicate logic for pruning backups against different storages

* Only parse templates when notifications are enabled

* Use better description
2022-02-13 10:52:19 +01:00
Frederik Ring
3e17d1b123
Ensure end time is recorded for unsuccessful runs too (#62)
* Ensure end time is also recorded for unsuccessful runs

* Clean up integration tests
2022-02-13 09:41:36 +01:00
Mauro Molin
8dfdd14527
Added custom notification messages using text/template (#60)
* Added custom notification messages using text/template

* Change notification template path and removed automatic newline trim

* Added stats and changed structure of template params

* Stat file hotfix

* Embedded and fixed default notification templates


Fix

* Changed Output to LogOutput

* Changed stats integer to unsigned

* Bytes formatting in template func


fix

* Changed Archives to Storages

* Removed unecessary sleep for pruning leeway

* Set EndTime after pruning is completed

* Added custom notifications documentation

* Added 5s sleep in swarm test

* Fixed documentation

* Dockerfile copies all files in cmd/backup
2022-02-11 20:05:16 +01:00
Frederik Ring
6fe629ce87
Allow path to be set for bucket storage (#52) 2022-01-25 21:16:16 +01:00
Frederik Ring
1db896f7cf Tweak README, improve client naming, tidy go.mod file 2022-01-22 13:35:13 +01:00
Kaerbr
6ded00aa06
Support Nextcloud / WebDav (#48)
* add studio-b12/gowebdav to be able to upload to webdav server

* make sure all env variables are present for webdav upload

* implement file upload to WebDav server

directory defaults to the base directory

* docs: add the new feature to the documentation

* if no WebDav env variable are given throw no error

* docs: use more elegant english :D

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* docs: use official spelling of "WebDAV"

* perf: golang likes to return early instead of having an else block

* use WEBDAV_PATH instead of WEBDAV_DIRECTORY

* use split_words for more convenience

like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support

* simplify

* feat: add pruning of files in WebDAV remote

Based on / Inspired by the minio/S3 implementation of pruning remote files.

* remove logging from the development

* test: first try implementing tests

Sandly I have to use the remote pipeline -- local wont work for me.

* test: adapt used volume names

* test: specify image only once!

* test: minio AND webdav data should be present

* test: backups on WebDAV remote are laying in the root directory

* test: the webdav server stores date in /var/lib/dav

* trying with data subfolder

* test: 1 container was added so the number raised from 3 to 4

* webdav  subfolder is "data" not "backup"

* fix: password AND username must be defined

not password OR username

* improve logging

* feat: if the given path on the server isnt preset it will be created

* test: add creation of new folder for webdav to tests

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2022-01-22 13:29:21 +01:00
Frederik Ring
32575c831e Also expand env vars in pruning prefix if configured 2021-12-23 09:22:56 +01:00
Frederik Ring
c062710ce8
Allow for env substitution in backup filename (#39) 2021-12-22 14:39:46 +01:00
Frederik Ring
4207146fb6 Refactor calling of hooks on exit 2021-12-18 10:31:12 +01:00
Frederik Ring
1f727f698f Run hooks in order of severity 2021-12-18 10:31:12 +01:00
Frederik Ring
88c90a206c Use int comparison for checking hooks 2021-12-18 10:31:12 +01:00
Frederik Ring
8bad0656b3 Enable notifications on multiple levels 2021-12-18 10:31:12 +01:00
Frederik Ring
08d78a0bd6 allow sending notifications to multiple channels 2021-12-18 10:31:12 +01:00
Frederik Ring
5a6ce81b58 update github.com/otai/copy, use PreserveOwner option 2021-11-29 08:40:55 +01:00
Frederik Ring
7bc5b2ccef fix minor error scoping mistakes 2021-11-28 20:06:24 +01:00
Frederik Ring
210c7d4540
Reuse hook mechanism for scheduling clean up tasks (#33)
* reuse hook mechanism for scheduling clean up tasks

* register hooks before creating files or dirs

* fix logging order

* use typed hook levels
2021-11-08 19:10:10 +01:00
schwannden
411c39ee72
create a snapshot before creating tar archive (#32)
* create a snapshot before creating tar archive

* safeguard snapshot removal and make snapshot optional

* fix typo, make sure remove snapshot failure triggers failure hook

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-11-08 08:39:18 +01:00
Frederik Ring
0c666d0c88 use lstat when checking whether file is a symlink 2021-11-03 18:07:55 +01:00
Frederik Ring
a0402b407d fix fileinfo mode comparison when checking for symlinks 2021-11-03 18:03:44 +01:00
Frederik Ring
3193e88fc0 os.FileInfo cannot be used for deleting files as it does not contain a full path 2021-11-02 06:40:37 +01:00
Frederik Ring
f946f36fb0 exclude symlinks from candidates when pruning local files
Previously, symlinks would be included in the set of candidates, but would
be skipped when pruning. This could lead to a wrong number of candidates
being printed in the log messages.
2021-10-29 09:00:37 +02:00
Frederik Ring
5245b5882f update README, save some indentation 2021-10-28 19:55:39 +02:00