db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-06-27 00:21:08 +02:00
Code Issues Releases Activity

Fix slow manifest build for very large quantities of tables/segments.

Browse Source 
storagePosixInfoList() processed each directory in a single memory context.  If the directory contained hundreds of thousands of files processing became very slow due to the number of allocations.

Instead, reset the memory context every thousand files to minimize the number of allocations active at once, improving both speed and memory consumption.

Reported by Jens Wilke.
This commit is contained in:
David Steele
2019-08-19 21:36:01 -04:00
parent d411321d28
commit 9eaeb33c88
2 changed files with 14 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
doc/xml/release.xml
View File

@ -14,6 +14,16 @@
<release-list>
<release date="XXXX-XX-XX" version="2.17dev" title="UNDER DEVELOPMENT">
<release-core-list>
<release-bug-list>
<release-item>
<release-item-contributor-list>
<release-item-ideator id="jens.wilke"/>
</release-item-contributor-list>
<p>Fix slow manifest build for very large quantities of tables/segments.</p>
</release-item>
</release-bug-list>
<release-improvement-list>
<release-item>
<release-item-contributor-list>

5
src/storage/posix/storage.c
View File

@ -223,7 +223,7 @@ storagePosixInfoList(THIS_VOID, const String *path, StorageInfoListCallback call
TRY_BEGIN()
{
MEM_CONTEXT_TEMP_BEGIN()
MEM_CONTEXT_TEMP_RESET_BEGIN()
{
// Read the directory entries
struct dirent *dirEntry = readdir(dir);
@ -235,6 +235,9 @@ storagePosixInfoList(THIS_VOID, const String *path, StorageInfoListCallback call
// Get next entry
dirEntry = readdir(dir);
// Reset the memory context occasionally so we don't use too much memory or slow down processing
MEM_CONTEXT_TEMP_RESET(1000);
}
}
MEM_CONTEXT_TEMP_END();