db/pg_probackup
1
0
Fork 0
mirror of https://github.com/postgrespro/pg_probackup.git synced 2026-06-21 01:34:15 +02:00
Code Issues Releases Activity
3,931 Commits 217 Branches 72 Tags
REL_2_6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yura Sokolov 649d1adc27 [PBCKP-431] refix test_minrecpoint_on_replica 
in fact there were too many wal_keep_segments set by default in
make_simple_node
 2022-12-29 05:37:39 +03:00
Yura Sokolov 361b9a7b51 fix for XLogReader's read_page callback contract 2022-12-29 04:13:36 +03:00
Yura Sokolov 3dd5219e13 try to fix test_minrecpoint_on_replica for Pg<=12 2022-12-29 04:12:45 +03:00
Yura Sokolov 428d10ddc6 pioDirNext: skip just deleted file 2022-12-29 03:26:49 +03:00
Yura Sokolov 0edd95862f s/test_needs_gdb/need_gdb/ 
fixtures plugin fails to fill 'func' fixture for 'test_needs_gdb' :-(
 2022-12-29 01:37:24 +03:00
Yura Sokolov 7b587f75f0 [PBCKP-423] and again 2022-12-28 13:44:16 +03:00
Yura Sokolov cf10582d6a [PBCKP-423] remove redundant ptrack warning 2022-12-28 13:30:04 +03:00
Yura Sokolov b2cf9afb0d @test_needs_gdb 
- make function decorator
- check for decorator in GDBObj
- check for linux's ptrace
 2022-12-28 12:31:28 +03:00
Yura Sokolov f38ac06872 reset timeouts in test_archive_replica_not_null_offset 2022-12-28 09:54:53 +03:00
Yura Sokolov 77fc12ac87 gdb kill did quit 2022-12-27 14:50:31 +03:00
Yura Sokolov f8b480e7c6 try to fix test_archive_get_batching_sanity 2022-12-27 14:44:41 +03:00
Yura Sokolov f5c0980d1c try to fix test_archive_replica_null_offset 2022-12-27 13:32:45 +03:00
Yura Sokolov cb19d6a24b fix replica_tests stop_lsn_null_offset 2022-12-27 13:19:25 +03:00
Yura Sokolov 03e5dce768 fix pioLocalReadFile_pioRead 2022-12-27 08:56:35 +03:00
Yura Sokolov dab39571f2 copy-paste some test fixes 
authors: Victoria Shepard and Sergey Fukanchik
 2022-12-26 13:27:33 +03:00
Yura Sokolov 0cf3d050f6 [PBCKP-434] make stop_backup_lsn volatile. 
Well, it is not really enough. We must use atomic. Lets do it later.
 2022-12-26 12:28:17 +03:00
Yura Sokolov ed609d8a3c again test_backup_with_less_privileges_role 
Try to delay switch_wal_segment to stabilize
 2022-12-26 12:25:03 +03:00
Yura Sokolov ecf0a7de40 [PBCKP-434] prefer to RunXLogThreads with consistent_read=true 
I doubt 'false' is ever usable.
 2022-12-26 11:59:41 +03:00
Yura Sokolov bed69c3ba4 [PBCKP-434] fix replica_test.py 
There are a lot less messages to check for.
And in fact backup in test_archive_replica_not_null_offset starts to
succeed.
 2022-12-26 11:59:41 +03:00
Yura Sokolov bc0ea49df6 [PBCKP-434] stabilize backup_test 
there was some nose-to-nose timing, so it could sporadically fail.
And in fact timeouts need no to be so large. Reduce them a bit.
 2022-12-26 11:59:41 +03:00
Yura Sokolov 8952b84b83 [PBCKP-434] greatly simplify stop_lsn handling 
It was long believed `stop_lsn` should point on start of last record.
But in fact it points to end of last record (because XLogInsert(BACKUP_END)
returns pointer to end of record, and that is what pg_backup_stop/
pg_stop_backup returns).

So we don't need to lookup `stop_lsn` record in WAL log, but rather check
existence of record which ends on `stop_lsn` (which is what
`get_prior_record_lsn` does).

And there is no need to raise error on `stop_lsn` which points on block end
- it is very valid when xlog record ends at block end.

So:
- we simplify wait_wal_lsn to just wait for `get_prior_record_lsn` returns
  "ok" (ie find such previous record). But now we don't overwrite stop_lsn
  with lsn of this record.
- we use `wait_wal_lsn` only in "ARCHIVE" mode/directory.
  So get rid of `in_stream_dir` argument.
- and `wait_wal_lsn` now waits only for previous record even for
  `is_start_lsn`, since there is no much gain in waiting for it.
  Even for PAGE mode we need only records before `start_lsn`, since
  `start_lsn` record is considered as part of backup.
- "STREAM" mode now waits for lsn by hooking streamer. We don't need to
  calculate stop_backup_lsn since we may use stop_backup_result.lsn
  directly to stop streamer.
  After streamer stopped, we just check wal with `get_prior_record_lsn`. We
  don't need to, but just for sanity.
- therefore `wait_wal_and_calculate_stop_lsn` become obsolete. Lets get rid
  of it.
  Without it, there is no need in get_first_record_lsn and
  get_next_record_lsn. Lets delete them too.
- Instead of using XRecOffIsValid and XRecOffIsNull use added
  XRecPtrLooksGood and XRecEndLooksGood.
  XRecPtrLooksGood validates record start more rigidly than XRecOffIsValid,
  and XRecEndLooksGood validates record end, which could be on block end.
- In fact, some of `XRecOffIsValid(x)` had to be `!XLogRecPtrIsInvalid(x)`
- Since stop_lsn is not inclusive now, RunXLogThreads doesn't need
  `inclusive_endpoint` parameter.
  (And because Page mode needs just till start_lsn)
- PAGE_LSN_FROM_FUTURE must be considered as error, btw.

(And by the way there's no need to XLogBeginRead before XLogFindNextRecord.
 Cleanup such calls)
 2022-12-26 11:59:41 +03:00
Yura Sokolov 79b8755b62 fix test_checkdb_checkunique 2022-12-26 11:59:03 +03:00
Yura Sokolov dc9cfb86b3 fix pioRemoteDrive_pioOpenRewrite 2022-12-25 19:24:59 +03:00
Yura Sokolov b0182bd5ed fix test_pgpro560_control_file_loss 2022-12-25 03:57:51 +03:00
Yura Sokolov 6a9850be62 fix test_options_5 2022-12-25 03:52:01 +03:00
Yura Sokolov 21c811c87a [PBCKP-365] jentlier 2022-12-25 03:50:25 +03:00
Yura Sokolov ffc8566097 [PBCKP-428] read_recovery_info don't need XRecOffIsValid(stop_lsn) 
since it uses XLogFindNextRecord, it could use stop_lsn with no valid
offset. In fact, it uses truncated stop_lsn with invalid offset always.

Also fix getting timestamp from possible record in a future.
We could not to not enter the loop since we need record->xl_prev to
correctly walk backward. So we need to add condition before
getRecordTimestamp.
 2022-12-24 04:57:10 +03:00
Yura Sokolov 12b8bfbc4c try to speedup test_replica_archive_page_backup 2022-12-24 04:45:02 +03:00
Yura Sokolov 4ddf8a1dfe speedup test_archive_replica_not_null_offset 2022-12-24 04:06:23 +03:00
Yura Sokolov cfcba44f97 [PBCKP-330] fix test_archive_replica_not_null_offset 
postgrespro enterprise has dirrerent xlog record size due to 64bit xid.
ee15 changed this size again.
 2022-12-24 03:23:55 +03:00
Yura Sokolov b70d76ec4a fix warning 2022-12-23 17:25:17 +03:00
Yura Sokolov c3430a685f [PBCKP-270] fetch current log position directly from streamer. 
We have streamer in our hand, so we may ask it directly, what are it
streaming now.

But in case we looks into previous segment, we fallback to log reading
since it is already flushed.
 2022-12-23 17:11:37 +03:00
Yura Sokolov 3e75df6db5 [PBCKP-270] fixes 2022-12-23 16:24:32 +03:00
Yura Sokolov 4418ec9572 [PBCKP-428] fix "optimize read_recovery_info" 
Well, we couldn't easily step back be 1MB, since it could be in
uninitialized tail of previous segment.

We need to use valid pointer in record->xl_prev.
 2022-12-23 12:31:43 +03:00
Yura Sokolov fe82f4e45e test_recovery_target_lsn_backup_victim - looks like it should pass 
and not fail
 2022-12-23 09:20:42 +03:00
Yura Sokolov c83d69c359 fix test_continue_failed_merge_2 2022-12-23 09:00:51 +03:00
Yura Sokolov f9615aeb6b fix test_merge_external_dir_is_missing 2022-12-23 08:47:56 +03:00
Yura Sokolov d8c41ae1eb fix init tests 2022-12-23 08:41:53 +03:00
Yura Sokolov a394ea3a5d fix test_missing_replication_permission_1 : fix freing xlogFile 
xlogFile did contain pgFile in its start. Therefore pgFileFree were used
to free it.

Not xlogFile doesn't contain pgFile, so its free procedure should be fixed.
 2022-12-23 08:18:02 +03:00
Yura Sokolov eecba5f4aa fix test_remove_instance_config 2022-12-23 08:04:50 +03:00
Yura Sokolov 46517a2fa0 fix test_backup_concurrent_drop_table 2022-12-23 07:59:15 +03:00
Yura Sokolov 80caef108b fix auth_test.test_backup_via_unprivileged_user 2022-12-23 07:47:22 +03:00
Yura Sokolov cc82beecd1 [PBCKP-428] optimize read_recovery_info 
Instead of stepping back by one record, step back by megabyte and then go
forward. This is much better both for gzipped wal and for S3.
 2022-12-23 07:46:49 +03:00
Yura Sokolov 6b8d3df368 [PBCKP-427] cache header map files 2022-12-23 06:49:03 +03:00
Yura Sokolov 7eed384934 [PBCKP-427] introduce pgFile hashtable 
We want to sort files be offset in page header map file.
But it breaks searching with binary search by filename.
Therefore make and use hashtable instead of binary search.
 2022-12-23 06:48:16 +03:00
Yura Sokolov c675380503 backup: descending sort by file size 
it is better suited for load balancing among threads
 2022-12-23 03:24:43 +03:00
Yura Sokolov be6ceac93a [PBCKP-426] use pioLocalFile in pioLocalPagesIterator 
I believe it better handles buffering with seeks than FILE*.
 2022-12-23 03:05:22 +03:00
Yura Sokolov d0b5e19bb9 [PBCKP-426] add special method pioFileStat for pioLocalReadFile 
It will be useful in pioLocalPagesIterator.
 2022-12-23 02:56:27 +03:00
Yura Sokolov 06326900a5 [PBCKP-426] buffer pioLocalReadFile 2022-12-23 02:50:49 +03:00
Yura Sokolov dd20305c5a [PBCKP-426] rename pioLocalFile->pioLocalReadFile 
just for symmetry with pioLocalWriteFile
 2022-12-23 01:15:23 +03:00
Yura Sokolov da99c02582 speedup pgpro434_3 and pgpro434_4 a bit 2022-12-22 17:35:15 +03:00
Yura Sokolov 7b375352ee improvement to ptrack_helpers.GDBobj 
autoclose it __del__
 2022-12-22 17:34:05 +03:00
Yura Sokolov 86609c2d30 archive.c: allow to overwrite broken gz file 2022-12-22 15:31:52 +03:00
Yura Sokolov a10d19a640 more complete send of error through remote connection 2022-12-22 15:31:52 +03:00
Yura Sokolov bc7a88b92f ability to alloc completely custom error 2022-12-22 15:31:52 +03:00
Yura Sokolov 67b406cc2b adapt do_validate_all to pio: use pioOpenDir 2022-12-22 06:20:50 +03:00
Yura Sokolov 4e4e76da07 Merge branch 'REL_2_5' into REL_2_6 2022-12-22 06:06:30 +03:00
Yura Sokolov 8f4e7d6e5f [PBCKP-346] archive-get doesn't need -D/--pgdata argument at all 2022-12-22 06:05:51 +03:00
Yura Sokolov dfc7f0466b fix show command: use pioOpenDir in catalog_get_instance_list 2022-12-22 05:35:37 +03:00
Yura Sokolov 1da2b77f92 remove fio_sync 2022-12-22 05:00:03 +03:00
Yura Sokolov 90cd7250a2 use pioSyncTree in restore_chain 2022-12-22 04:57:39 +03:00
Yura Sokolov d69902ddce merge.c: use temp files and rename from pioOpenRewrite 
though we had to add sync arguments trough backup_data_file and
backup_non_data_file down to send_pages and send_file.
 2022-12-22 04:57:39 +03:00
Yura Sokolov ffdd4b3cf7 use pioSyncTree in catchup_sync_destination_files 2022-12-22 04:57:39 +03:00
Yura Sokolov dfade6a5d5 use pioSyncTree in do_backup_pg 
and remove sync parameter from pioOpenRewrite in send_pages
 2022-12-22 03:52:33 +03:00
Yura Sokolov eb5ee89880 pioSyncTree - sync whole subtree 2022-12-22 03:24:53 +03:00
Yura Sokolov f8d010b21d rename pio_recursive_dir_t -> pioRecursiveDir 
pioRecursiveDir is really an object, so pio_recursive_dir_free were quite
misleading.
 2022-12-22 01:36:43 +03:00
Yura Sokolov d15d34bdb5 fix setting LC_TIME 2022-12-22 01:16:30 +03:00
Yura Sokolov 54602f1a55 set LC_TIME to C as well 
private branch uses strptime to parse "always C locale" time.
 2022-12-22 00:44:36 +03:00
Yura Sokolov efb5cf7793 ... 2022-12-21 23:16:14 +03:00
Yura Sokolov 76bbd0401b ... 2022-12-21 10:58:38 +03:00
Yura Sokolov d29a8d4c54 fix "Cleanup CFS breadcrumbs" 2022-12-21 08:19:20 +03:00
Yura Sokolov d50c60b7db fix build with Pg < 12 2022-12-21 07:56:01 +03:00
Yura Sokolov 098b5c6b35 artificial main function in main.c file 
To introduce C unit tests we separate `main` function into distinct file,
so we may have `main` function in test files.
 2022-12-20 12:10:59 +03:00
Yura Sokolov 01c42a4655 remove couple of FIO_LOCAL_HOST 2022-12-20 05:32:48 +03:00
Yura Sokolov b7b71eef72 make pio_helper_pioGetCRC32 to be useful for other pioDrive implementations 2022-12-20 05:22:14 +03:00
Yura Sokolov f91b2256e0 remove all non-pio CRC32 staff + some additional cleanup 2022-12-20 05:03:45 +03:00
Yura Sokolov 5164bb1ea0 remove fio_get_crc32 from restore_non_data_file 2022-12-20 04:55:20 +03:00
Yura Sokolov 5898a66c39 huge fio_location->pioDrive_i refactor (still partial) 
I've started with attempt to remove fio_get_crc32 from backup_non_data_file.
It led to huge replacement from fio_location to pioDrive_i.
I didn't want it :-( But it should be done nevertheless.
 2022-12-20 04:47:16 +03:00
Yura Sokolov 9673496b47 make pioGetCRC32 to use pioCRC32Counter 2022-12-20 02:59:01 +03:00
Yura Sokolov 7836066053 validate.c: remove direct usage of pgFileGetCRC32C 2022-12-20 02:29:10 +03:00
Yura Sokolov 4cf6270f99 refactor pgFileNew: now it calculates crc by itself 
it will reduce calls to pgFileGetCRC32C as well
 2022-12-20 02:10:02 +03:00
Yura Sokolov 89b28fe885 remove unused trick in fio_send_file_content_impl 2022-12-20 01:35:01 +03:00
Yura Sokolov 557df03504 remove some no more used staff 2022-12-20 01:31:13 +03:00
Yura Sokolov 7d48f6f620 fix test_page_doesnt_store_unchanged_cfm 2022-12-20 01:18:47 +03:00
Yura Sokolov 54aabfdabb format previous commit a bit 2022-12-20 00:52:36 +03:00
Yura Sokolov e43dfb47a2 do_init: use pioIsDirEmpty for existance check 2022-12-19 06:05:25 +03:00
Yura Sokolov 44c0d3874a fixup "make do_add_instance more pio friendly" 2022-12-19 05:24:46 +03:00
Yura Sokolov 39de9b8b18 use pioExists(expected_kind = PIO_KIND_DIRECTORY) to check instance path 2022-12-19 05:22:58 +03:00
Yura Sokolov 50b262fbfe make do_add_instance more pio friendly 2022-12-19 05:16:02 +03:00
Yura Sokolov 81b8d0c23e fix ft_strbuf_cat_path 2022-12-19 04:57:44 +03:00
Yura Sokolov e86535e6a0 more remote connection ensuring 2022-12-19 03:48:42 +03:00
Yura Sokolov 43d48110c3 fix test_no_control_file 2022-12-19 03:48:42 +03:00
Yura Sokolov 719cce1b1b ft_str_chop1 - cut 1 byte from string (+ fix ft_str_ends_withc return type) 2022-12-19 03:48:42 +03:00
Yura Sokolov 92495b85d2 [PBCKP-416] PIO_DIR_NEXT: abuse empty entry as stop signal. 2022-12-19 03:48:42 +03:00
Yura Sokolov b4d2d821f2 ft_strbuf_cat_zt - fix for empty string 2022-12-19 03:48:42 +03:00
Yura Sokolov fd3fba7894 [PBCKP-416] db_list_dir: reuse stat from dir iteration 2022-12-19 03:48:42 +03:00
Yura Sokolov 07137a715b [PBCKP-416] remove fio_opendir/fio_readdir/fio_closedir 2022-12-19 03:48:42 +03:00
Yura Sokolov 19c33c9240 [PBCKP-416] fix test_basic_missing_dir_permissions 2022-12-19 03:48:42 +03:00
Yura Sokolov ff779a6e5f [PBCKP-416] rewrite db_list_dir with pio_recursive_dir 2022-12-19 03:48:42 +03:00
Yura Sokolov 7d552c6046 [PBCKP-416] pio_recursive_dir iterator. 
Iterator that could recursively walk through dir hierarchy and skip some
subdirectories.
 2022-12-19 03:48:42 +03:00