db/pg_probackup
1
0
Fork 0
mirror of https://github.com/postgrespro/pg_probackup.git synced 2026-06-21 01:34:15 +02:00
Code Issues Releases Activity
3,840 Commits 217 Branches 72 Tags
78360660534c9fb7c969668658f140c8fc02feed
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yura Sokolov 7836066053 validate.c: remove direct usage of pgFileGetCRC32C 2022-12-20 02:29:10 +03:00
Yura Sokolov 4cf6270f99 refactor pgFileNew: now it calculates crc by itself 
it will reduce calls to pgFileGetCRC32C as well
 2022-12-20 02:10:02 +03:00
Yura Sokolov 89b28fe885 remove unused trick in fio_send_file_content_impl 2022-12-20 01:35:01 +03:00
Yura Sokolov 557df03504 remove some no more used staff 2022-12-20 01:31:13 +03:00
Yura Sokolov 7d48f6f620 fix test_page_doesnt_store_unchanged_cfm 2022-12-20 01:18:47 +03:00
Yura Sokolov 54aabfdabb format previous commit a bit 2022-12-20 00:52:36 +03:00
Yura Sokolov e43dfb47a2 do_init: use pioIsDirEmpty for existance check 2022-12-19 06:05:25 +03:00
Yura Sokolov 44c0d3874a fixup "make do_add_instance more pio friendly" 2022-12-19 05:24:46 +03:00
Yura Sokolov 39de9b8b18 use pioExists(expected_kind = PIO_KIND_DIRECTORY) to check instance path 2022-12-19 05:22:58 +03:00
Yura Sokolov 50b262fbfe make do_add_instance more pio friendly 2022-12-19 05:16:02 +03:00
Yura Sokolov 81b8d0c23e fix ft_strbuf_cat_path 2022-12-19 04:57:44 +03:00
Yura Sokolov e86535e6a0 more remote connection ensuring 2022-12-19 03:48:42 +03:00
Yura Sokolov 43d48110c3 fix test_no_control_file 2022-12-19 03:48:42 +03:00
Yura Sokolov 719cce1b1b ft_str_chop1 - cut 1 byte from string (+ fix ft_str_ends_withc return type) 2022-12-19 03:48:42 +03:00
Yura Sokolov 92495b85d2 [PBCKP-416] PIO_DIR_NEXT: abuse empty entry as stop signal. 2022-12-19 03:48:42 +03:00
Yura Sokolov b4d2d821f2 ft_strbuf_cat_zt - fix for empty string 2022-12-19 03:48:42 +03:00
Yura Sokolov fd3fba7894 [PBCKP-416] db_list_dir: reuse stat from dir iteration 2022-12-19 03:48:42 +03:00
Yura Sokolov 07137a715b [PBCKP-416] remove fio_opendir/fio_readdir/fio_closedir 2022-12-19 03:48:42 +03:00
Yura Sokolov 19c33c9240 [PBCKP-416] fix test_basic_missing_dir_permissions 2022-12-19 03:48:42 +03:00
Yura Sokolov ff779a6e5f [PBCKP-416] rewrite db_list_dir with pio_recursive_dir 2022-12-19 03:48:42 +03:00
Yura Sokolov 7d552c6046 [PBCKP-416] pio_recursive_dir iterator. 
Iterator that could recursively walk through dir hierarchy and skip some
subdirectories.
 2022-12-19 03:48:42 +03:00
Yura Sokolov 2c756c3397 Revert "[PBCKP-234] proper pioListDir usage" partially 
This reverts commit 4ecb11f45d.

It seems it would be easier to change to pioOpenDir with rollback.
But we keep pioRemoveDir usage.
 2022-12-19 03:48:42 +03:00
Yura Sokolov dfb5b0dc0d [PBCKP-416] pioLocalDrive_pioRemoveDir to not use pioListDir 
yep, this way code is larger. But it self contained.
 2022-12-19 03:48:42 +03:00
Yura Sokolov 65fe3d032d [PBCKP-416] use pioOpenDir in archive.c setup_push_filelist 2022-12-19 03:48:42 +03:00
Yura Sokolov f30c94a192 [PBCKP-416] remove backup_list_dir 2022-12-19 03:48:42 +03:00
Yura Sokolov 0d145dac21 [PBCKP-416] use pioOpenDir in catalog_get_timelines 2022-12-19 03:48:42 +03:00
Yura Sokolov f626bba162 [PBCKP-416] pioIsDirEmpty 2022-12-18 21:32:44 +03:00
Yura Sokolov 776b3b5f23 ... 2022-12-18 21:32:44 +03:00
Yura Sokolov 2df87a0e14 [PBCKP-416] catalog_get_backup_list - use pioOpenDir/pioDirNext 2022-12-18 21:32:44 +03:00
Yura Sokolov 17badb0c26 Revert "PBCKP-415 use pio for catalog_get_backup_list" 
This reverts commit f0215bbcc4.
 2022-12-18 21:32:44 +03:00
Yura Sokolov e79e5f0728 [PBCKP-416] pioOpenDir + pioDirNext 
Looks like pioListDir were no so great idea. Step to return to iterator
style.
 2022-12-18 21:32:44 +03:00
Yura Sokolov abdfdd2777 ft_strbuf_cat_path 2022-12-18 21:32:21 +03:00
Yura Sokolov d2843d0a11 fix for fio_receive_pio_err 2022-12-18 21:32:15 +03:00
Yura Sokolov 5341b9d0a7 remove pioAsyncWrite/pioAsyncError 2022-12-18 21:32:04 +03:00
Yura Sokolov 133c2275d5 fix cleanup_tablespace: it is for DB, not for BACKUP 2022-12-18 16:39:32 +03:00
Yura Sokolov 04c3159edf ft_utils: more useful staff. 2022-12-18 16:39:32 +03:00
Yura Sokolov 6bfdc0b024 adapt fobj missing argument compile-time error generation 
And it actually found error in data.c.
 2022-12-18 01:33:58 +03:00
Yura Sokolov bce53bbd31 fu_util/fobj: make missing argument compilation error 2022-12-18 01:33:38 +03:00
Yura Sokolov 33f7c10d2d fix clang warning about format(gnu_printf) 2022-12-18 01:33:38 +03:00
Yura Sokolov d0d06f3e91 fix ft_strbuf_ensure for zero-initialized ft_strbuf_t 2022-12-18 01:32:08 +03:00
Yura Sokolov 47de7fea75 ft_util: easier to parse and compose from C structs 
FT_BYTES_FOR - bytes buffer for variable
ft_bytes_dup - alloc copy of bytes
ft_bytes_shift_to/shift_must - cut to bytes from the beginning.
 2022-12-18 01:31:56 +03:00
Yura Sokolov 2b274c7a6a move pioRename from pioDrive to pioDBDrive 
it has single invocation in archive.c to move "<WAL>.ready" to "<WAL>.done"
on database host.
 2022-12-17 19:47:45 +03:00
Yura Sokolov 0852bf9abe [PBCKP-417] move sync paramter from pioClose to pioOpenWrite/Rewrite 
pioClose was not so comfortable place for. We always know at the file
opening will it be sync or not.
 2022-12-17 02:21:37 +03:00
Yura Sokolov eaead56049 use ft_strbuf_catc_zt + ft_bytes_shift_zt 2022-12-17 01:34:17 +03:00
Yura Sokolov 61c4d90316 ft_strbuf_cat_zt and ft_bytes_shift_str to easier remote operations 2022-12-17 01:28:19 +03:00
Yura Sokolov 81e2f7fe75 [PBCKP-416] do_catchup calls db_list_dir with backup_logs = false. 2022-12-16 23:22:14 +03:00
Yura Sokolov 68113e38d8 fu_utils: add test for correct thread termination 2022-12-16 02:20:49 +03:00
Yura Sokolov f8fbbbbed3 ... 2022-12-16 01:57:41 +03:00
Yura Sokolov 47e21044c8 Merge branch 'REL_2_5' into REL_2_6 2022-12-15 22:52:12 +03:00
Yura Sokolov 764680e23f Merge branch 'REL_2_5' 2022-12-15 22:47:46 +03:00
Yura Sokolov b84bed3ecb fix Makefile 2022-12-15 19:01:03 +03:00
Yura Sokolov dcf90d689c merge.c: unused buffer 2022-12-15 18:59:16 +03:00
Yura Sokolov 401bccf0b7 fu_util/fobj: fix autorelease pool cleanup in glibc 
glibc's pthread_exit runs stack unwinding for C++ destructor's handling,
and calls destructors for pthread specifics only after unwinding.

This way fobj_destroy_thread_AR is called too lately.

We have to compile with '-fexception' to register all our
attribute((cleanup)) for autorelease pools as "C++ desctructors".

Found in debug session with Sergey Fucanchik <s.fukanchik@postgrespro.ru>
 2022-12-15 18:57:40 +03:00
Yura Sokolov 8efe4d4bc7 remove fio fd-style functions and gzFile-style functions as unused 2022-12-15 11:13:15 +03:00
Yura Sokolov 692f0c0e7b SimpleXlogPageRead: replace fileExists with pioExists 2022-12-15 11:02:01 +03:00
Yura Sokolov d231fc6be5 fix pioReadFilter_pioRead 
1. without this change it attempt to another one pioRead on wrapped even if
  we already filled wbuf.
2. this lead to strange decompression error in SimpleXlogPageRead. I didn't
  investigate it properly, but this change fixes things.
 2022-12-15 11:02:01 +03:00
Yura Sokolov d175255933 use pioOpenRead and pioWrapForReSeek in SimpleXlogPageRead 2022-12-15 11:02:01 +03:00
Yura Sokolov fdab5a9149 prepare to replace fio in parsexlog.c: seekable decompressor 
Seekable decompressor is a cheat: read from underlying file starts from
beginning if we seek backward.

But it is best we can do as gzseek does the same.
 2022-12-15 10:33:11 +03:00
Yura Sokolov d3ad1f4318 remove a lot of unused fio functions 2022-12-15 07:33:34 +03:00
Yura Sokolov fdda848511 [PBCKP-404] use pioCopy instead of restore_non_data_file_internal 2022-12-15 07:04:12 +03:00
Yura Sokolov ad178a80d0 [PBCKP-404] use pioReader in restore_data_file and validate_file_pages 
reorganize common code in those functions into backup_page_iterator.
 2022-12-15 07:04:12 +03:00
Yura Sokolov 36335b6047 remove pioReadFull 
looks like all pioRead methods already tries to read all possible data
 2022-12-15 05:45:34 +03:00
Yura Sokolov f8286da568 [PBCKP-404] use pioDBWriter for restore_data_file/restore_non_data_file 2022-12-15 03:46:41 +03:00
Yura Sokolov 482c5d9933 fix pioTruncate/pioSeek signature: it is better to accept uint64_t 2022-12-15 01:01:40 +03:00
Yura Sokolov f3a5de561a merge_test: duplicate fio_remove breakpoints to pioRemove__do 2022-12-14 16:13:25 +03:00
Yura Sokolov 3f18ffab8b fix pioWriteCompressed 2022-12-14 15:59:22 +03:00
Yura Sokolov fceeb0abf2 Merge branch 'REL_2_5' into REL_2_6 2022-12-14 11:12:37 +03:00
Yura Sokolov 1f773d6d54 change pioWrite signature since it always return error on short write 
and we don't use short write in any meaningful way.
 2022-12-14 11:09:33 +03:00
Yura Sokolov 42ef001b9d add pioWriteCompressed to pioDBWriter 2022-12-14 10:55:42 +03:00
Yura Sokolov 7fa6986306 [PBCKP-395] split pioOpen to pioOpenRead and pioOpenStream 
it simplifies pioCopy as well
 2022-12-14 09:59:46 +03:00
Yura Sokolov ddb0d3329d [PBCKP-395] force async erorr fetching in pioRemoteWriteFile_pioClose 2022-12-14 09:26:01 +03:00
Yura Sokolov 36e52ff9ab fix for pioLocalFileWriter 2022-12-14 09:19:09 +03:00
Yura Sokolov e25f8ab1cc [PBCKP-395] intermediate step: forbid writting on pgFile 
All write activity is moved to pioOpenWrite and pioOpenRewrite.
pgFile will be renamed to pgReader and pgReadStream.
 2022-12-14 08:59:28 +03:00
Yura Sokolov 55fbf0b954 [PBCKP-395] use pioOpenWrite in data.c copy_pages 2022-12-14 08:42:56 +03:00
Yura Sokolov 662833bb37 [PBCKP-395] use pioOpenWrite in archive.c get_wal_file 2022-12-14 08:42:56 +03:00
Yura Sokolov 2428adb7a5 [PBCKP-395] and looks pioDBWriter needs pioTruncate 2022-12-14 08:42:22 +03:00
Yura Sokolov 9e1f00de79 [PBCKP-395] add pioDBDrive.pioOpenWrite for opening file with pioSeek 
pioSeek is used in restorations, therefore we have to have it.
But it is not usable for S3, there it is not in pioDrive.
 2022-12-14 08:17:47 +03:00
Yura Sokolov 5e82af7f3a [PBCKP-395] use pioOpenRewrite in create_empty_file 2022-12-14 08:12:13 +03:00
Yura Sokolov 644d5f72c8 [PBCKP-395] use pioOpenRewrite in archive.c: push_file_internal 
As a unfortunate side effect it leads to absence of concurrent archiving
detection. This way "last-win" happens.

S3 gives no way to solve the issue (and this change is preparatino for S3).
But we could use file-locking if storage is on-disk. So push_file_internal
should be re-improved to use it.

test_archive_push_part_file_exists_not_stale is disabled and should be
rewritten to accompany rewritten push_file_internal with file-locking.

test_archive_push_partial_file_exists is removed completely.
 2022-12-14 08:12:13 +03:00
Yura Sokolov ac4e509a82 [PBCKP-395] use pioOpenRewrite in pioLocalDrive_pioWriteFile 2022-12-14 08:12:13 +03:00
Yura Sokolov 85dc5c09a0 [PBCKP-395] use pioOpenRewrite in write_backup_filelist 2022-12-14 08:12:13 +03:00
Yura Sokolov 5021a99657 [PBCKP-395] use pioOpenWrite in send_pages 2022-12-14 08:12:13 +03:00
Yura Sokolov 19fcad262e [PBCKP-395] implement remote pioOpenRewrite 
pioRemoteWriteFile uses async api exclusively.
 2022-12-14 08:12:13 +03:00
Yura Sokolov 096a8f9fe0 [PBCKP-395] introduce pioLocalDrive_pioOpenRewrite 
This is "only streaming write" file abstraction with support for
atomicity. It uses temporary file and rename on-disk storage.
 2022-12-14 08:12:13 +03:00
Yura Sokolov 156419422c expose pioDBDrive more 2022-12-14 08:12:13 +03:00
Yura Sokolov c516fcb7b9 change pioSeek signature 2022-12-14 07:36:45 +03:00
Yura Sokolov 3e3178e567 ... pioWrite will return error if it writes lesser. 2022-12-14 07:36:45 +03:00
Yura Sokolov 698824e581 file.c: regularize remote handle handling 2022-12-14 06:53:57 +03:00
Yura Sokolov 7866628420 fo_obj.h: improve debug-ability by making method entry function static 2022-12-13 19:17:34 +03:00
Yura Sokolov 1d2faa383b fo_obj: hide warnings in $iref/$iunref 2022-12-13 16:56:34 +03:00
Yura Sokolov 38f59c6567 fix pioWriteFilter_pioWriteFinish 2022-12-13 16:23:10 +03:00
Yura Sokolov 10ac3c9918 [PBCKP-402] bound check for abortedRecPtr to Pg15 
It is not really needed in previous versions.
It doesn't harm, but we want to reduce tests amount.
 2022-12-13 13:09:34 +03:00
Yura Sokolov 24f12d98d9 [PBCKP-402,PBCKP-354] refix "missing contrecord" detection. 
Error message is translated according to current locale. So we can't
compare it as a string. But `abortedRecPtr` exists exactly for this case,
so we can rely on it.
 2022-12-13 02:52:18 +03:00
Yura Sokolov e683cf8a5c fix compilation for < PG12 2022-12-12 12:49:20 +03:00
Yura Sokolov 77b47e20d7 remove fetch.c because functions defined here are unused 2022-12-10 16:29:30 +03:00
Yura Sokolov 36ff719717 and another one unused function 2022-12-09 10:21:42 +03:00
Yura Sokolov 6898075741 remove some already unused functions 2022-12-09 10:15:38 +03:00
Yura Sokolov e3c322a926 buffer for pioLocalPagesIterator 2022-12-09 10:08:57 +03:00
Yura Sokolov 5dfb164d2a send_pages: simplify headers handling 2022-12-09 00:55:26 +03:00
Yura Sokolov 2b387ed853 remove datapagemap_iterator and optimize datapatemap_first 2022-12-09 00:42:05 +03:00