You've already forked pigallery2
mirror of
https://github.com/bpatrik/pigallery2.git
synced 2025-10-08 22:52:02 +02:00
let exifr read the file insteadof the app guessing the size. #277
This commit is contained in:
Patrik J. Braun
@@ -1,510 +0,0 @@
|
||||
# PiGallery2 Design: Projection Search Context, Search Sharing, and User-level Allow/Blacklist
|
||||
|
||||
Author: Junie (JetBrains autonomous programmer)
|
||||
Date: 2025-08-13
|
||||
Status: Updated after maintainer clarifications
|
||||
|
||||
Terminology: The project uses the term projection. Earlier drafts and filenames may still say “projected”; whenever you see “projected,” read it as “projection.” Benchmark file names keep “projected” for historical reasons.
|
||||
|
||||
## Overview
|
||||
This document proposes a design to support:
|
||||
- Search-based sharing: a share link represents a pre-filtered view of the gallery.
|
||||
- User-level allow AND blacklist via search query: a user’s session is constrained by an allow list and/or deny list expressed as search filters.
|
||||
|
||||
Both features depend on a common capability: a “projection” that prefilters the whole gallery and propagates to all dependent features (directories, persons, counts, covers, etc.). The solution must be efficient on low-end devices (e.g., Raspberry Pi), support ~10 concurrent users, and up to ~100 concurrent sharings.
|
||||
|
||||
Maintainer decisions incorporated:
|
||||
- User supports both allow and blacklist via search query.
|
||||
- Path-based sharing is removed; sharing a path must be expressed as a strict directory search query.
|
||||
- Covers for a projection are selected by recursively scanning the directory and all descendants for the best matching media, and the selected cover must be persisted in the DB per projection.
|
||||
- Prefer DB persistence for caching derived/aggregated values; avoid in-memory caches where possible. The DB acts as the cache; media files are the source of truth.
|
||||
- The UI search query (search bar) is NOT part of the Projection Context. Apply it only to media listing/search; derived aggregates (covers, persons) ignore it. User Allow/Blacklist and Share projection always apply.
|
||||
- UI filters are local-only (filter.service.ts) and never travel to the backend; do not include them in backend projection.
|
||||
- SQLite and MySQL must be supported.
|
||||
|
||||
## Current Architecture Summary (as observed)
|
||||
- Database/ORM: TypeORM. Entities live under src/backend/model/database/enitites/ (typo in folder name).
|
||||
- Core entities impacted by projection:
|
||||
- MediaEntity (base), PhotoEntity, VideoEntity
|
||||
- DirectoryEntity: holds derived fields mediaCount, oldestMedia, youngestMedia, validCover, cover
|
||||
- PersonEntry with PersonJunctionTable: person counts and sample face derived from present media
|
||||
- FileEntity: meta files (markdown/gpx/pg2conf)
|
||||
- SharingEntity: currently path-based sharing with optional password and expiry
|
||||
- album/SavedSearchEntity: persistent named searches
|
||||
- Search logic is encapsulated in SearchManager, which builds complex TypeORM queries from a SearchQueryDTO. It implements:
|
||||
- Multiple query primitives (text, people, rating, resolution, date ranges, orientation, etc.)
|
||||
- Query flattening and combination helpers (AND/OR/SomeOf)
|
||||
|
||||
## Requirements
|
||||
1. Search sharing
|
||||
- Generate a link that encapsulates a search query (or references a saved search).
|
||||
- Guest session is restricted to the pre-filtered subset of media and derived aggregates (directories, persons, counts, covers).
|
||||
- Path-based shares are deprecated/removed; to share a folder use a strict directory search query (see below).
|
||||
2. User allow and blacklist via query
|
||||
- A per-user static filter AllowQuery and/or DenyQuery expressed in the same query language.
|
||||
- Effective user projection = AllowQuery AND NOT DenyQuery.
|
||||
3. Common: All derived values must reflect the projection
|
||||
- Directory inferred values (mediaCount, youngest/oldestMedia, cover) must be computed with respect to projection.
|
||||
- Persons list, person counts, sample faces must only include media within projection.
|
||||
- Prefer DB-persisted caches for derived values.
|
||||
4. Scale targets: ~10 users, up to 100 sharings.
|
||||
|
||||
## Entity Dependencies on Photos (Media)
|
||||
- DirectoryEntity depends on MediaEntity for:
|
||||
- mediaCount
|
||||
- oldestMedia, youngestMedia (timestamps from contained media)
|
||||
- cover, validCover
|
||||
- PersonEntry depends on PersonJunctionTable which references MediaEntity (face regions are tied to specific media). PersonEntry.count and sampleRegion are derived from available media.
|
||||
- SavedSearchEntity references a SearchQueryDTO (JSON). Its results are entirely dependent on the set of Media.
|
||||
- SharingEntity will be extended to hold a searchQuery (JSON). Share content depends on Media and the query.
|
||||
- FileEntity (meta) is not directly dependent on Photos, but its availability in a directory often tracks the presence of media.
|
||||
|
||||
## Design
|
||||
### 1) Projection Context (Projection-based Search Context)
|
||||
Introduce a request-level object that represents the effective filter for the session, called ProjectionContext. It is computed from the following inputs:
|
||||
- UserProjection: AllowQuery AND NOT DenyQuery (if configured for the user)
|
||||
- ShareProjection: Share.searchQuery (if present)
|
||||
|
||||
EffectiveProjection = UserProjection AND ShareProjection
|
||||
|
||||
Notes:
|
||||
- Local UI filters (frontend/filter.service.ts) and the UI search bar query are client-only for transient listing and must not be included in ProjectionContext or its caches.
|
||||
|
||||
Representation:
|
||||
- ProjectionContext.raw: the three inputs
|
||||
- ProjectionContext.effective: a canonicalized, flattened SearchQueryDTO usable by SearchManager
|
||||
- ProjectionContext.signature (projectionKey): a stable hash (e.g., SHA-256) of the canonicalized effective query. Used as a cache key, persisted with aggregates.
|
||||
|
||||
Integration patterns (two viable implementations):
|
||||
A. Filtered repositories (injection-based)
|
||||
- Provide wrapper Repository/Manager for Media, Directory, Person that automatically injects base predicates into QueryBuilder calls.
|
||||
|
||||
B. Projection Query Builders (composition-based)
|
||||
- Provide ProjectionQB factory:
|
||||
- getFilteredMediaQB(): SelectQueryBuilder<MediaEntity>
|
||||
- getFilteredDirectoryQB(): SelectQueryBuilder<DirectoryEntity> (joins to filtered media)
|
||||
- getFilteredPersonsQB(): SelectQueryBuilder<PersonEntry> (joins PersonJunctionTable -> filtered media)
|
||||
- Managers receive ProjectionContext and must obtain QBs from the factory.
|
||||
|
||||
Given PiGallery2’s current QueryBuilder-heavy approach in SearchManager, approach B is recommended initially (lower-risk, incremental). We can adopt A later if desired.
|
||||
|
||||
Implementation detail: Query reuse
|
||||
- Build the base filtered media subquery once per request: baseFMQ = SearchManager.prepareAndBuildWhereQuery(ProjectionContext.effective)
|
||||
- Reuse baseFMQ as a subquery/CTE in subsequent directory/person queries to ensure consistency and minimize re-parsing.
|
||||
|
||||
### 2) Derived/Inferred Values under a Projection (Persisted)
|
||||
For projection views we compute per-projection values and persist them in DB as cache tables keyed by ProjectionContext.signature (projectionKey). Client DTOs MUST be composed by merging:
|
||||
- Base (unprojected) entity data: directory/album name, path, ids, etc.
|
||||
- Projection aggregates: item counts, oldest/youngest timestamps, and cover.
|
||||
|
||||
Deprecation note: Existing unprojected derived columns on DirectoryEntity (mediaCount, oldestMedia, youngestMedia, validCover, cover) should no longer be used by readers. New code paths must consume projection caches. A follow-up migration can drop these columns once all reads are updated.
|
||||
|
||||
Proposed cache entities (SQLite and MySQL compatible):
|
||||
- ProjectionDirectoryCacheEntity (merged: directory aggregates + cover)
|
||||
- projectionKey TEXT (hash of canonicalized effective query)
|
||||
- directoryId INT (FK -> DirectoryEntity)
|
||||
- mediaCount INT
|
||||
- oldestMedia BIGINT (nullable)
|
||||
- youngestMedia BIGINT (nullable)
|
||||
- coverMediaId INT (nullable, FK -> MediaEntity)
|
||||
- projectionCoverValid BOOLEAN DEFAULT 0
|
||||
- Unique(projectionKey, directoryId)
|
||||
- ProjectionAlbumCacheEntity (album aggregates + cover)
|
||||
- projectionKey TEXT
|
||||
- albumId INT (FK -> AlbumBaseEntity)
|
||||
- itemCount INT
|
||||
- oldestMedia BIGINT (nullable)
|
||||
- youngestMedia BIGINT (nullable)
|
||||
- coverMediaId INT (nullable, FK -> MediaEntity)
|
||||
- projectionCoverValid BOOLEAN DEFAULT 0
|
||||
- Unique(projectionKey, albumId)
|
||||
- ProjectionPersonAggEntity
|
||||
- projectionKey TEXT
|
||||
- personId INT (FK -> PersonEntry)
|
||||
- count INT
|
||||
- sampleRegionId INT (nullable, FK -> PersonJunctionTable)
|
||||
- Unique(projectionKey, personId)
|
||||
|
||||
Computation strategy
|
||||
- On read: try cache row(s) for the given projectionKey; if missing, compute via aggregated SQL using baseFMQ and upsert rows.
|
||||
- We avoid global content-version-based invalidation. Instead, use targeted invalidation:
|
||||
- When a directory’s content changes and DirectoryEntity.validCover is reset, also reset projectionCoverValid = 0 for that directory across all projection rows.
|
||||
- When media is added/removed in a directory, lazily recompute that directory’s ProjectionDirectoryCacheEntity row on next access (upsert overwrites).
|
||||
- Upsert mechanics:
|
||||
- SQLite: INSERT INTO ... ON CONFLICT(projectionKey, directoryId) DO UPDATE SET ...
|
||||
- MySQL: INSERT ... ON DUPLICATE KEY UPDATE ... (with a composite unique index)
|
||||
|
||||
### 3) Cover Selection (Recursive) and Persistence
|
||||
- Selection rule: For a given directory D and projection, search for best matching media in D and all of its subdirectories recursively.
|
||||
- Ranking: Use Config.AlbumCover.Sorting. Prefer media from D itself over descendants where applicable (as currently implemented with CASE sorting), then apply sorting.
|
||||
- Persistence: Upsert into ProjectionDirectoryCacheEntity for (projectionKey, directoryId) with coverMediaId and set projectionCoverValid = 1. This avoids recomputing on subsequent reads.
|
||||
- Fallback: If no media found, persist coverMediaId = NULL and projectionCoverValid = 1 to avoid repeated scans (still keyed by projectionKey).
|
||||
- Invalidation: Reuse the existing cover invalidation semantics at directory-level. When a directory (or its subtree) content changes and DirectoryEntity.validCover is reset, also reset projectionCoverValid = 0 for the affected directory rows across projections. Avoid global invalidation.
|
||||
|
||||
### 4) Search Sharing (Path removed; strict directory share)
|
||||
Schema changes
|
||||
- SharingEntity: remove path; add non-nullable searchQuery (TEXT with JSON transformer).
|
||||
- A “share a path” use-case is expressed as a strict directory search query:
|
||||
- { type: SearchQueryTypes.directory, text: FullDirectoryPath, matchType: exact_match }
|
||||
- FullDirectoryPath means DirectoryEntity.path + DirectoryEntity.name (the unique directory path in PiGallery2).
|
||||
- Backward compatibility/migration: Existing path-based shares should be migrated by creating equivalent directory-exact searchQuery records.
|
||||
|
||||
API
|
||||
- POST /api/share: accepts { searchQuery, password?, valid } only. No path parameter.
|
||||
- GET using sharingKey establishes ShareProjection from the stored searchQuery.
|
||||
|
||||
Security
|
||||
- sharingKey remains unguessable; password/expiry unchanged.
|
||||
|
||||
### 5) User-level Allow AND Blacklist via Search Query
|
||||
Schema extension
|
||||
- UserEntity: add allowQuery (nullable TEXT JSON), denyQuery (nullable TEXT JSON).
|
||||
- Effective user projection: allowQuery AND NOT denyQuery; if allowQuery is null, it means TRUE (unrestricted); if denyQuery is null, it means FALSE (no deny).
|
||||
|
||||
Operational notes
|
||||
- Load and attach these queries at login to construct ProjectionContext for each request.
|
||||
|
||||
### 6) Combining Projections and Precedence
|
||||
- User projection: Allow AND NOT Deny
|
||||
- Share projection: Share.searchQuery
|
||||
|
||||
EffectiveProjection = UserProjection ∧ ShareProjection
|
||||
|
||||
### 7) Performance Considerations (Raspberry Pi friendly)
|
||||
- Prefer single aggregated SQL queries per view over iterative per-entity fetches.
|
||||
- Reuse the base filtered media subquery in all aggregations to avoid duplicative parsing and planning.
|
||||
- Ensure indices exist on common predicates:
|
||||
- Media: directoryId, creationDate, rating, orientation, resolution
|
||||
- PersonJunctionTable: mediaId, personId
|
||||
- DirectoryEntity: parentId, path, name
|
||||
- Add indices/unique constraints for cache tables on (projectionKey, directoryId)/(projectionKey, personId).
|
||||
- For SQLite: GLOB-based path recursion is already in use; for MySQL use LIKE prefix as implemented today.
|
||||
|
||||
### 8) Invalidation Strategy
|
||||
- No time-based cleanup on projection caches. Do not depend on updatedAt/TTL or content-version keys.
|
||||
- Explicit invalidation only:
|
||||
- When a directory’s content changes and DirectoryEntity.validCover is reset, also reset projectionCoverValid = 0 for that directory across all projection rows (ProjectionDirectoryCacheEntity).
|
||||
- Directory aggregates and album aggregates are recomputed lazily on next read: perform the aggregate query and upsert the row for the given projectionKey and entity id (overwrite existing values).
|
||||
- Person aggregates are recomputed lazily on demand similarly.
|
||||
- The only periodic cleanup remains existing expired sharing deletion (unchanged).
|
||||
|
||||
### 9) API/Code Integration Plan
|
||||
- Introduce ProjectionContext builder utility that:
|
||||
1) Canonicalizes, flattens, AND-combines UserProjection and ShareProjection
|
||||
2) Builds the base filtered media subquery via SearchManager
|
||||
3) Exposes projectionKey (signature) for cache tables
|
||||
4) Note: UI searchQueryDTO is NOT part of ProjectionContext; apply it separately to media listing/search only (never included in projectionKey).
|
||||
- Update managers to:
|
||||
- Read from cache tables by projectionKey; if miss, compute and upsert.
|
||||
- Use recursive cover selection when populating ProjectionDirectoryCacheEntity.
|
||||
- Server endpoints that accept a searchQueryDTO from the UI apply it only to media listing/search; do NOT include it in ProjectionContext or projectionKey.
|
||||
- Sharing routes: accept only searchQuery; remove path parameter(s).
|
||||
- User sessions: load user’s allow/deny at login and use for ProjectionContext.
|
||||
|
||||
### 10) Testing Strategy
|
||||
- Unit tests for ProjectionContext combination and canonicalization.
|
||||
- Integration tests:
|
||||
- Directory listing under a projection: counts, oldest/youngest, covers reflect only filtered media and are persisted in caches.
|
||||
- Persons list under a projection: set membership and counts correct; persisted.
|
||||
- Search sharing: guest views match query, unauthorized media excluded; strict directory shares behave non-recursively.
|
||||
- User allow/deny: ensure deny overrides, and effective results equal Allow AND NOT Deny.
|
||||
- Performance tests on low-spec environment settings, validating response times under target concurrency with modest dataset.
|
||||
|
||||
### 11) Migration Considerations
|
||||
- Path removal in SharingEntity:
|
||||
- Add searchQuery column (TEXT JSON) and drop path.
|
||||
- Migrate existing rows: for each share(path=p), write searchQuery = directory-exact query for p.
|
||||
- Add cache tables: ProjectionDirectoryCacheEntity, ProjectionAlbumCacheEntity, and ProjectionPersonAggEntity, and indices. Add composite unique index on PersonJunctionTable(mediaId, personId) to avoid duplicates and improve joins.
|
||||
- Optional follow-up migration: drop unprojected derived columns from DirectoryEntity (mediaCount, oldestMedia, youngestMedia, validCover, cover) once all reads use projection caches; client DTOs will continue to merge base identity with projection aggregates, remaining transparent to the client.
|
||||
|
||||
### 12) PersonJunctionTable and Person Queries
|
||||
- Current schema: PersonJunctionTable has id (PK), and ManyToOne relations to MediaEntity (media) and PersonEntry (person), both indexed. This supports joins in both directions.
|
||||
- Use cases supported:
|
||||
- List all persons under a projection: join pj -> media (filtered by baseFMQ) -> group by pj.personId, count rows; optionally pick a sampleRegionId from pj linked media in-projection.
|
||||
- List all photos for a given person under a projection: join pj on pj.personId = :id; join media; filter by baseFMQ; select media.
|
||||
- Recommendations:
|
||||
- Add a composite UNIQUE index on (mediaId, personId) to prevent accidental duplicates and improve planner efficiency.
|
||||
- Ensure single-column indices on mediaId and personId (already present via @Index on relations in the entity).
|
||||
- No additional schema changes are required; coordinates of faces remain in MediaEntity.metadata.faces (JSON), while PersonEntry.sampleRegion can continue referencing a pj row.
|
||||
- SearchManager notes:
|
||||
- Person text search currently relies on Media.metadata.persons (simple-array) and personsLength for min/max. This can remain for text search performance. Aggregations (person lists, counts) should use the junction table joined to the filtered media set for correctness under a projection.
|
||||
|
||||
## Appendix: Pseudocode Snippets
|
||||
|
||||
Build ProjectionContext (exclude UI search query; exclude local UI filters):
|
||||
```
|
||||
function buildProjectionContext(user, sharing): ProjectionContext {
|
||||
const allowQ = user.allowQuery ?? TRUE;
|
||||
const denyQ = user.denyQuery ? NOT(user.denyQuery) : TRUE;
|
||||
const userProj = AND(allowQ, denyQ);
|
||||
const shareQ = sharing?.searchQuery ?? TRUE;
|
||||
const effective = AND(userProj, shareQ);
|
||||
const canonical = canonicalize(flatten(effective));
|
||||
const projectionKey = hash(canonical);
|
||||
return { raw: {allowQ, denyQ, shareQ}, effective: canonical, signature: projectionKey };
|
||||
}
|
||||
```
|
||||
|
||||
Recursive cover for a directory and projection (simplified TypeORM-ish):
|
||||
```
|
||||
const fmSubQ = searchManager.prepareAndBuildWhereQuery(projection.effective);
|
||||
const q = repo.createQueryBuilder('media')
|
||||
.innerJoin('media.directory', 'directory')
|
||||
.where(/* directory.id == dirId OR directory.path under dir path, DB-specific */)
|
||||
.andWhere(`media.id IN (${fmSubQ.getQuery()})`)
|
||||
.select(['media.id'])
|
||||
.orderBy(`CASE WHEN directory.id = :dirId THEN 0 ELSE 1 END`, 'ASC')
|
||||
.setParameters({ dirId })
|
||||
/* then apply Config.AlbumCover.Sorting */
|
||||
.limit(1);
|
||||
const coverMedia = await q.getOne();
|
||||
UPSERT ProjectionDirectoryCacheEntity(projectionKey, dirId) SET coverMediaId = (coverMedia?.id || NULL), projectionCoverValid = 1;
|
||||
```
|
||||
|
||||
Directory aggregates (persisted on miss):
|
||||
```
|
||||
const fmSubQ = ...;
|
||||
const agg = await repo.createQueryBuilder('directory')
|
||||
.leftJoin(MediaEntity, 'fm', 'fm.directoryId = directory.id')
|
||||
.andWhere(`fm.id IN (${fmSubQ.getQuery()})`)
|
||||
.select('directory.id', 'directoryId')
|
||||
.addSelect('COUNT(fm.id)', 'mediaCount')
|
||||
.addSelect('MIN(fm.metadata.creationDate)', 'oldestMedia')
|
||||
.addSelect('MAX(fm.metadata.creationDate)', 'youngestMedia')
|
||||
.groupBy('directory.id')
|
||||
.getRawMany();
|
||||
UPSERT all rows into ProjectionDirectoryCacheEntity with projectionKey;
|
||||
```
|
||||
|
||||
Persons under a projection (persisted on miss):
|
||||
```
|
||||
const fmSubQ = ...;
|
||||
const rows = await repo.createQueryBuilder('p')
|
||||
.leftJoin(PersonJunctionTable, 'pj', 'pj.personId = p.id')
|
||||
.leftJoin(MediaEntity, 'm', 'm.id = pj.mediaId')
|
||||
.andWhere(`m.id IN (${fmSubQ.getQuery()})`)
|
||||
.select('p.id', 'personId')
|
||||
.addSelect('COUNT(pj.id)', 'count')
|
||||
.groupBy('p.id')
|
||||
.getRawMany();
|
||||
UPSERT rows into ProjectionPersonAggEntity with projectionKey;
|
||||
```
|
||||
|
||||
---
|
||||
End of document.
|
||||
|
||||
|
||||
## Appendix: Projection Cache Schema Benchmark (Denormalized vs Normalized projectionKey)
|
||||
|
||||
Purpose
|
||||
- Compare two designs for projection-cache tables used by the EffectiveProjection caching approach:
|
||||
- Variant A (current in doc): cache tables store projectionKey TEXT directly with composite unique keys.
|
||||
- Variant B (alternative): separate ProjectionKey table (unique key hash), projection tables reference it via projectionId (FK ON DELETE CASCADE).
|
||||
|
||||
Benchmark harness
|
||||
- Location: benchmark\\projected-cache-bench.ts (historical filename)
|
||||
- Run: npm run bench:projected-cache
|
||||
- Params (env vars): BENCH_SCOPES, BENCH_DIRS, BENCH_PERSONS, BENCH_LOOKUPS. Example (Windows cmd):
|
||||
- set BENCH_SCOPES=20&&set BENCH_DIRS=120&&set BENCH_PERSONS=90&&set BENCH_LOOKUPS=1000&&npm run bench:projected-cache
|
||||
- DB: temporary SQLite file at db\\projected_bench.sqlite (does not touch app DB)
|
||||
- Measures: upsert throughput for dir/person tables, lookup latency by (projection, directory), cascade delete performance, file size delta.
|
||||
|
||||
Sample results (SQLite, N=20, D=120, P=90, lookups=1000)
|
||||
- Insert/Upsert
|
||||
- A.upsert ProjDirA: 188.79 ms
|
||||
- A.upsert ProjPersonA: 101.38 ms
|
||||
- B.upsert ProjDirB: 203.97 ms (~+8%)
|
||||
- B.upsert ProjPersonB: 117.96 ms (~+16%)
|
||||
- Lookup (find one by projection + directory)
|
||||
- A.lookup ProjDirA: 268.46 ms
|
||||
- B.lookup ProjDirB: 798.40 ms (~3x)
|
||||
- Deletion
|
||||
- A.delete projection rows by projectionKey (2 tables): 89.12 ms
|
||||
- B.delete ProjectionKey rows (cascade): 48.90 ms (~45% faster)
|
||||
- File size delta: 0 bytes (SQLite files do not shrink without VACUUM).
|
||||
|
||||
Notes and interpretation
|
||||
- Normalizing projection keys (Variant B) significantly improves deletion/invalidation thanks to ON DELETE CASCADE (single delete on ProjectionKey), supporting our targeted invalidation strategy.
|
||||
- Writes (upserts) are modestly slower with normalization on this dataset.
|
||||
- Lookups in Variant B were slower in this run; likely factors:
|
||||
- ORM overhead when referencing relation columns.
|
||||
- Need to ensure composite unique index on (projectionId, directoryId) is used by queries. The benchmark defines a composite UNIQUE which should create an index; however, query-path differences (QueryBuilder vs Repository) can influence timings.
|
||||
- For fairness, align query shapes and add an explicit @Index(["projection", "directoryId"]) if needed; also consider prepared statements.
|
||||
- MySQL not measured here; normalization tends to pay off further with larger datasets and cascading cleanups. The harness can be extended to a MySQL DataSource if desired.
|
||||
|
||||
Recommendation (initial)
|
||||
- If fast and simple invalidation across many projections is a priority, prefer Variant B (ProjectionKey + FK cascade) and ensure:
|
||||
- Composite unique index on (projectionId, directoryId) and (projectionId, personId).
|
||||
- Query by both columns to leverage the index.
|
||||
- If lookup performance dominates and invalidation is infrequent, Variant A may be slightly faster in SQLite.
|
||||
- Given PiGallery2’s need to invalidate many projection rows cheaply (covers and aggregates), Variant B looks favorable, provided we optimize lookups and add proper indices. The benchmark harness is now available to iterate on these optimizations.
|
||||
|
||||
|
||||
## Implementation Plan (Variant A: projectionKey TEXT)
|
||||
|
||||
Decision
|
||||
- We will implement projection caches using Variant A: projectionKey stored as TEXT in cache tables with composite unique indices. This aligns with maintainer preference (“keeping projectionKey: text”) and provides good lookup performance and simpler code paths. The earlier benchmark remains in the appendix for reference; we may revisit normalization (Variant B) later if invalidation complexity grows.
|
||||
|
||||
Audience
|
||||
- This plan is written so a junior engineer can follow it step by step. Each task lists files to touch, acceptance criteria, and notes.
|
||||
|
||||
High-level milestones
|
||||
1) Schema: add projection cache tables (directory aggregates + cover merged, person aggregates).
|
||||
2) Projection builder: construct EffectiveProjection and projectionKey.
|
||||
3) Cache compute & read‑through: populate caches on demand using SearchManager.
|
||||
4) Invalidation: reuse existing cover invalidation; add targeted resets for projection caches.
|
||||
5) Sharing/User projection: move share to searchQuery; wire user allow/deny.
|
||||
6) Managers integration: Gallery and Person readers consume caches.
|
||||
7) Tests and rollout.
|
||||
|
||||
Prerequisites
|
||||
- Node v22 (per engines), local DB (SQLite default) available.
|
||||
- Ability to run npm scripts: npm run build-backend, npm run test-backend.
|
||||
- Familiarity with TypeORM entities and QueryBuilder (see SearchManager.ts for patterns).
|
||||
|
||||
Task 1: Add new cache entities (schema)
|
||||
- Files to create:
|
||||
- src\\backend\\model\\database\\enitites\\cache\\ProjectionDirectoryCacheEntity.ts
|
||||
- src\\backend\\model\\database\\enitites\\cache\\ProjectionAlbumCacheEntity.ts
|
||||
- src\\backend\\model\\database\\enitites\\cache\\ProjectionPersonAggEntity.ts
|
||||
- Content (fields and constraints):
|
||||
- ProjectionDirectoryCacheEntity
|
||||
- projectionKey: TEXT (indexed)
|
||||
- directoryId: INT, FK -> DirectoryEntity (indexed)
|
||||
- mediaCount: INT
|
||||
- oldestMedia: BIGINT nullable
|
||||
- youngestMedia: BIGINT nullable
|
||||
- coverMediaId: INT nullable, FK -> MediaEntity
|
||||
- projectionCoverValid: BOOLEAN DEFAULT 0
|
||||
- Unique(projectionKey, directoryId)
|
||||
- ProjectionAlbumCacheEntity
|
||||
- projectionKey: TEXT (indexed)
|
||||
- albumId: INT, FK -> AlbumBaseEntity (indexed)
|
||||
- itemCount: INT
|
||||
- oldestMedia: BIGINT nullable
|
||||
- youngestMedia: BIGINT nullable
|
||||
- coverMediaId: INT nullable, FK -> MediaEntity
|
||||
- projectionCoverValid: BOOLEAN DEFAULT 0
|
||||
- Unique(projectionKey, albumId)
|
||||
- ProjectionPersonAggEntity
|
||||
- projectionKey: TEXT (indexed)
|
||||
- personId: INT, FK -> PersonEntry (indexed)
|
||||
- count: INT
|
||||
- sampleRegionId: INT nullable, FK -> PersonJunctionTable
|
||||
- Unique(projectionKey, personId)
|
||||
- Integration:
|
||||
- Add all new entities to SQLConnection.getEntries() list so TypeORM can create tables.
|
||||
- Acceptance criteria:
|
||||
- Running the app with synchronize=false and schemeSync flow creates the tables in SQLite/MySQL without errors (dev DB).
|
||||
|
||||
Task 2: Build ProjectionContext utility (EffectiveProjection and projectionKey)
|
||||
- Files to create:
|
||||
- src\\backend\\model\\ProjectionContext.ts
|
||||
- Responsibilities:
|
||||
- Inputs: user.allowQuery, user.denyQuery, sharing.searchQuery; exclude local UI filters and the UI search bar query.
|
||||
- Combine as EffectiveProjection = AND(Allow, NOT(Deny), Share).
|
||||
- Canonicalize/flatten queries (reuse SearchManager.flattenSameOfQueries where possible; keep stable ordering of AND/OR lists).
|
||||
- Compute projectionKey = SHA‑256 of canonicalized EffectiveProjection JSON string.
|
||||
- Expose helper to obtain base filtered media Brackets via SearchManager.prepareAndBuildWhereQuery(EffectiveProjection).
|
||||
- Acceptance criteria:
|
||||
- Unit tests cover: combining inputs, stability of projectionKey for semantically identical trees, exclusion of local UI filters and the UI search bar query.
|
||||
|
||||
Task 3: Projection cache read‑through and upsert (Directory & Albums)
|
||||
- Files to create:
|
||||
- src\\backend\\model\\database\\ProjectionCacheManager.ts (new manager)
|
||||
- Responsibilities (directory side):
|
||||
- getDirectoryAggregates(projectionKey, dirIds[]): read rows from ProjectionDirectoryCacheEntity; for misses, compute aggregates via a single aggregated query using base filtered media subquery, then upsert rows (ON CONFLICT/ON DUPLICATE KEY).
|
||||
- getAndPersistProjectionCover(projectionKey, directory): compute cover by recursively scanning directory + descendants with EffectiveProjection (reuse CoverManager sorting/rules), then upsert coverMediaId and set projectionCoverValid=1. If no media found, persist NULL with projectionCoverValid=1.
|
||||
- Responsibilities (album side):
|
||||
- getAlbumAggregates(projectionKey, albumIds[]): for each album, compute itemCount/oldest/youngest by querying Media with EffectiveProjection AND album.searchQuery (intersection); upsert into ProjectionAlbumCacheEntity.
|
||||
- getAndPersistProjectionAlbumCover(projectionKey, album): compute best cover with EffectiveProjection AND album.searchQuery (reuse CoverManager.getCoverForAlbum logic with extra filter); upsert coverMediaId and set projectionCoverValid=1 (or NULL with projectionCoverValid=1 if no match).
|
||||
- SQL patterns:
|
||||
- SQLite: INSERT INTO ... ON CONFLICT(projectionKey, directoryId|albumId) DO UPDATE SET ...
|
||||
- MySQL: INSERT ... ON DUPLICATE KEY UPDATE ...
|
||||
- Acceptance criteria:
|
||||
- Given a projection, repeated calls for the same directory/album return cached rows without recomputing; first call computes and persists.
|
||||
|
||||
Task 4: Projection cache read‑through and upsert (Persons)
|
||||
- Extend ProjectionCacheManager with:
|
||||
- getPersonsAggregates(projectionKey): list persons and counts within projection by joining PersonJunctionTable -> MediaEntity filtered by EffectiveProjection; upsert rows into ProjectionPersonAggEntity.
|
||||
- Optionally compute a sampleRegionId per person from an in‑projection face.
|
||||
- Acceptance criteria:
|
||||
- Listing persons under a projection yields correct counts; second call hits cache.
|
||||
|
||||
Task 5: Invalidation and maintenance
|
||||
- Directory cover invalidation:
|
||||
- When DirectoryEntity.validCover is reset (existing logic), also execute: UPDATE ProjectionDirectoryCacheEntity SET projectionCoverValid = 0 WHERE directoryId IN (affected dirs).
|
||||
- Media add/remove in a directory:
|
||||
- No global invalidation. On next read, aggregates recompute and overwrite via upsert (lazy refresh).
|
||||
- Acceptance criteria:
|
||||
- After adding/removing media in a directory, subsequent requests recompute aggregates on demand; projection cover recomputes once projectionCoverValid=0.
|
||||
|
||||
Task 6: Sharing changes (path removed; strict directory query)
|
||||
- Update DTOs (already present in src\\common\\entities\\SharingDTO.ts to include searchQuery). Ensure backend aligns:
|
||||
- src\\backend\\model\\database\\enitites\\SharingEntity.ts: remove path; add @Column('text', transformer: JSON<->SearchQueryDTO>) searchQuery.
|
||||
- src\\backend\\middlewares\\SharingMWs.ts: accept only { searchQuery, password, valid } in create/update; remove path; validation updated.
|
||||
- src\\backend\\model\\database\\SharingManager.ts: persist searchQuery; drop path logic.
|
||||
- src\\backend\\middlewares\\user\\AuthenticationMWs.ts: on share login, set session user allowList = sharing.searchQuery.
|
||||
- Migration:
|
||||
- For existing path shares, create equivalent directory‑exact SearchQuery and store to new column; drop path column.
|
||||
- Acceptance criteria:
|
||||
- Creating and fetching shares works with searchQuery only; “share a path” is done by providing a strict directory search query.
|
||||
|
||||
Task 7: User allow AND blacklist
|
||||
- Extend UserEntity (backend):
|
||||
- Add allowQuery (TEXT JSON, nullable), denyQuery (TEXT JSON, nullable).
|
||||
- Session wiring:
|
||||
- At login, load these fields and pass to ProjectionContext builder.
|
||||
- Acceptance criteria:
|
||||
- EffectiveProjection = Allow AND NOT Deny reflected in results; null means unrestricted/none as specified.
|
||||
|
||||
Task 8: Managers integration
|
||||
- GalleryManager:
|
||||
- When listing directories, use ProjectionCacheManager.getDirectoryAggregates() with current projectionKey to populate projection counts and times for visible directories.
|
||||
- For subdirectory tiles, call getAndPersistProjectionCover() if projectionCoverValid=0 or missing.
|
||||
- Albums:
|
||||
- Album list/detail endpoints should use ProjectionCacheManager.getAlbumAggregates() and getAndPersistProjectionAlbumCover() to populate itemCount/oldest/youngest and cover per projection.
|
||||
- CoverManager:
|
||||
- Expose a method to get recursive cover with an optional EffectiveProjection Brackets; used by ProjectionCacheManager.
|
||||
- Person listing endpoints:
|
||||
- Use ProjectionCacheManager.getPersonsAggregates() instead of ad‑hoc counts.
|
||||
- Acceptance criteria:
|
||||
- Directory/album/person views reflect projection aggregates; performance remains acceptable on Raspberry Pi targets.
|
||||
|
||||
Task 9: PersonJunctionTable index improvement
|
||||
- Add composite UNIQUE index on (mediaId, personId) in PersonJunctionTable to prevent duplicates and speed joins.
|
||||
- Acceptance criteria:
|
||||
- Schema contains the composite UNIQUE; queries using joins show stable performance.
|
||||
|
||||
Task 10: Testing
|
||||
- Unit tests:
|
||||
- ProjectionContext building and hashing stability.
|
||||
- Integration tests:
|
||||
- Directory listing under a projection: counts/oldest/youngest/cover reflect only filtered media and persist.
|
||||
- Person list under a projection: counts correct; persisted.
|
||||
- Share flow: searchQuery‑only shares function; strict directory share works.
|
||||
- User allow/deny: deny overrides; EffectiveProjection applies.
|
||||
- Performance check:
|
||||
- On sample dataset, verify latency does not regress; optional use of the existing benchmark harness.
|
||||
|
||||
Task 11: Deployment and rollback
|
||||
- Bump DataStructureVersion if existing schemeSync requires it to create new tables; verify no destructive resets in production. If risk exists, prepare a backup/migration plan.
|
||||
- Feature flags:
|
||||
- Optionally guard projection cache usage with a config flag to allow quick rollback to non‑projection behavior.
|
||||
- Rollback path:
|
||||
- If issues arise, disable projection caches (flag) and keep existing global behavior while investigating.
|
||||
|
||||
Checklist (Definition of Done)
|
||||
- [ ] New cache entities exist and are registered.
|
||||
- [ ] ProjectionContext utility builds EffectiveProjection and provides projectionKey.
|
||||
- [ ] ProjectionCacheManager computes and upserts directory aggregates and covers.
|
||||
- [ ] Persons aggregates persisted and served from cache.
|
||||
- [ ] Invalidation hooks wired to existing cover invalidation and directory changes.
|
||||
- [ ] Sharing uses searchQuery only; path removed; migration path documented.
|
||||
- [ ] User allow/deny supported and included in EffectiveProjection.
|
||||
- [ ] Managers consume projection caches; views reflect projection results.
|
||||
- [ ] Tests added and passing; basic performance verified on low‑spec device.
|
||||
|
||||
Notes
|
||||
- Keep paths Windows‑style (e.g., src\\backend\\model\\database\\enitites\\...).
|
||||
- For MySQL, ensure the composite UNIQUE indices exist and use VARCHAR for TEXT columns if needed for index limitations.
|
||||
@@ -1,347 +0,0 @@
|
||||
/*
|
||||
Scoped cache schema performance benchmark
|
||||
- Variant A: denormalized scopeKey TEXT in cache tables
|
||||
- Variant B: normalized ScopeKey table with FK (ON DELETE CASCADE)
|
||||
|
||||
Usage:
|
||||
npx ts-node benchmark\\scoped-cache-bench.ts
|
||||
|
||||
Notes:
|
||||
- Uses a temporary SQLite DB at db\\scoped_bench.sqlite
|
||||
- Does NOT touch the application DB
|
||||
*/
|
||||
|
||||
import 'reflect-metadata';
|
||||
import {DataSource, Entity, PrimaryGeneratedColumn, Column, Index, ManyToOne, JoinColumn, Unique, Repository} from 'typeorm';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
// -------------------- Entities (Variant A - denormalized) --------------------
|
||||
|
||||
@Entity({ name: 'scoped_dir_a' })
|
||||
@Unique(['scopeKey', 'directoryId'])
|
||||
class ScopedDirA {
|
||||
@PrimaryGeneratedColumn({ unsigned: true })
|
||||
id!: number;
|
||||
|
||||
@Index()
|
||||
@Column('text')
|
||||
scopeKey!: string;
|
||||
|
||||
@Index()
|
||||
@Column('int', { unsigned: true })
|
||||
directoryId!: number;
|
||||
|
||||
@Column('int', { unsigned: true })
|
||||
mediaCount!: number;
|
||||
|
||||
@Column('bigint', { nullable: true })
|
||||
oldestMedia!: number | null;
|
||||
|
||||
@Column('bigint', { nullable: true })
|
||||
youngestMedia!: number | null;
|
||||
|
||||
@Column('int', { unsigned: true, nullable: true })
|
||||
coverMediaId!: number | null;
|
||||
|
||||
@Column('bigint', { unsigned: true })
|
||||
updatedAt!: number;
|
||||
}
|
||||
|
||||
@Entity({ name: 'scoped_person_a' })
|
||||
@Unique(['scopeKey', 'personId'])
|
||||
class ScopedPersonA {
|
||||
@PrimaryGeneratedColumn({ unsigned: true })
|
||||
id!: number;
|
||||
|
||||
@Index()
|
||||
@Column('text')
|
||||
scopeKey!: string;
|
||||
|
||||
@Index()
|
||||
@Column('int', { unsigned: true })
|
||||
personId!: number;
|
||||
|
||||
@Column('int', { unsigned: true })
|
||||
count!: number;
|
||||
|
||||
@Column('int', { unsigned: true, nullable: true })
|
||||
sampleRegionId!: number | null;
|
||||
|
||||
@Column('bigint', { unsigned: true })
|
||||
updatedAt!: number;
|
||||
}
|
||||
|
||||
// -------------------- Entities (Variant B - normalized with FK) --------------------
|
||||
|
||||
@Entity({ name: 'scope_key_b' })
|
||||
@Unique(['key'])
|
||||
class ScopeKeyB {
|
||||
@PrimaryGeneratedColumn({ unsigned: true })
|
||||
id!: number;
|
||||
|
||||
@Index()
|
||||
@Column('text')
|
||||
key!: string;
|
||||
|
||||
@Column('bigint', { unsigned: true })
|
||||
createdAt!: number;
|
||||
}
|
||||
|
||||
@Entity({ name: 'scoped_dir_b' })
|
||||
@Unique(['scope', 'directoryId'])
|
||||
class ScopedDirB {
|
||||
@PrimaryGeneratedColumn({ unsigned: true })
|
||||
id!: number;
|
||||
|
||||
@ManyToOne(() => ScopeKeyB, { onDelete: 'CASCADE', nullable: false })
|
||||
@JoinColumn({ name: 'scopeId' })
|
||||
scope!: ScopeKeyB;
|
||||
|
||||
@Index()
|
||||
@Column('int', { unsigned: true })
|
||||
directoryId!: number;
|
||||
|
||||
@Column('int', { unsigned: true })
|
||||
mediaCount!: number;
|
||||
|
||||
@Column('bigint', { nullable: true })
|
||||
oldestMedia!: number | null;
|
||||
|
||||
@Column('bigint', { nullable: true })
|
||||
youngestMedia!: number | null;
|
||||
|
||||
@Column('int', { unsigned: true, nullable: true })
|
||||
coverMediaId!: number | null;
|
||||
|
||||
@Column('bigint', { unsigned: true })
|
||||
updatedAt!: number;
|
||||
}
|
||||
|
||||
@Entity({ name: 'scoped_person_b' })
|
||||
@Unique(['scope', 'personId'])
|
||||
class ScopedPersonB {
|
||||
@PrimaryGeneratedColumn({ unsigned: true })
|
||||
id!: number;
|
||||
|
||||
@ManyToOne(() => ScopeKeyB, { onDelete: 'CASCADE', nullable: false })
|
||||
@JoinColumn({ name: 'scopeId' })
|
||||
scope!: ScopeKeyB;
|
||||
|
||||
@Index()
|
||||
@Column('int', { unsigned: true })
|
||||
personId!: number;
|
||||
|
||||
@Column('int', { unsigned: true })
|
||||
count!: number;
|
||||
|
||||
@Column('int', { unsigned: true, nullable: true })
|
||||
sampleRegionId!: number | null;
|
||||
|
||||
@Column('bigint', { unsigned: true })
|
||||
updatedAt!: number;
|
||||
}
|
||||
|
||||
// -------------------- Bench helpers --------------------
|
||||
|
||||
function hrtimeMs(start?: bigint): number | bigint {
|
||||
const now = process.hrtime.bigint();
|
||||
if (!start) return now;
|
||||
return Number(now - start) / 1_000_000; // ms
|
||||
}
|
||||
|
||||
async function time<T>(label: string, fn: () => Promise<T>): Promise<{ label: string; ms: number; result: T }> {
|
||||
const s = hrtimeMs() as bigint;
|
||||
const result = await fn();
|
||||
const ms = hrtimeMs(s) as number;
|
||||
console.log(`${label}: ${ms.toFixed(2)} ms`);
|
||||
return { label, ms, result };
|
||||
}
|
||||
|
||||
function randInt(maxExclusive: number): number {
|
||||
return Math.floor(Math.random() * maxExclusive);
|
||||
}
|
||||
|
||||
async function upsertBatch(repo: Repository<any>, items: any[], conflictPaths: string[], chunk = 1000) {
|
||||
for (let i = 0; i < items.length; i += chunk) {
|
||||
const slice = items.slice(i, i + chunk);
|
||||
await repo.upsert(slice as any, conflictPaths as any);
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------- Main --------------------
|
||||
|
||||
(async () => {
|
||||
const dbFolder = path.join(process.cwd(), 'db');
|
||||
if (!fs.existsSync(dbFolder)) {
|
||||
fs.mkdirSync(dbFolder);
|
||||
}
|
||||
const dbPath = path.join(dbFolder, 'scoped_bench.sqlite');
|
||||
if (fs.existsSync(dbPath)) {
|
||||
fs.unlinkSync(dbPath);
|
||||
}
|
||||
|
||||
const ds = new DataSource({
|
||||
type: 'better-sqlite3',
|
||||
database: dbPath,
|
||||
entities: [ScopeKeyB, ScopedDirA, ScopedPersonA, ScopedDirB, ScopedPersonB],
|
||||
synchronize: true,
|
||||
dropSchema: true,
|
||||
});
|
||||
|
||||
await ds.initialize();
|
||||
|
||||
// Add FKs for cascade manually (TypeORM for better-sqlite3 sets them, but ensure PRAGMA is on)
|
||||
await ds.query('PRAGMA foreign_keys = ON');
|
||||
|
||||
const scopeRepoB = ds.getRepository(ScopeKeyB);
|
||||
const dirARepo = ds.getRepository(ScopedDirA);
|
||||
const personARepo = ds.getRepository(ScopedPersonA);
|
||||
const dirBRepo = ds.getRepository(ScopedDirB);
|
||||
const personBRepo = ds.getRepository(ScopedPersonB);
|
||||
|
||||
// Parameters (can be tweaked via env vars)
|
||||
const N = parseInt(process.env.BENCH_SCOPES || '100', 10); // scopes
|
||||
const D = parseInt(process.env.BENCH_DIRS || '500', 10); // directories
|
||||
const P = parseInt(process.env.BENCH_PERSONS || '300', 10); // persons
|
||||
const lookups = parseInt(process.env.BENCH_LOOKUPS || '10000', 10); // lookup ops
|
||||
|
||||
console.log('Benchmark params:', { N, D, P, lookups });
|
||||
|
||||
const scopeKeys: string[] = Array.from({ length: N }, (_, i) => `scope_${i}_${Math.random().toString(36).slice(2, 10)}`);
|
||||
|
||||
// Prepare ScopeKeyB rows
|
||||
await time('B.insert ScopeKey rows', async () => {
|
||||
const rows = scopeKeys.map((k) => {
|
||||
const r = new ScopeKeyB();
|
||||
r.key = k;
|
||||
r.createdAt = Date.now();
|
||||
return r;
|
||||
});
|
||||
await scopeRepoB.insert(rows);
|
||||
});
|
||||
|
||||
const scopeIdByKey = new Map<string, number>();
|
||||
{
|
||||
const rows = await scopeRepoB.find();
|
||||
rows.forEach((r) => scopeIdByKey.set(r.key, r.id));
|
||||
}
|
||||
|
||||
// Create data for Variant A
|
||||
await time('A.upsert ScopedDirA', async () => {
|
||||
const batch: ScopedDirA[] = [];
|
||||
const now = Date.now();
|
||||
for (let i = 0; i < N; i++) {
|
||||
for (let d = 1; d <= D; d++) {
|
||||
const row = new ScopedDirA();
|
||||
row.scopeKey = scopeKeys[i];
|
||||
row.directoryId = d;
|
||||
row.mediaCount = (d * 7 + i) % 50;
|
||||
row.oldestMedia = now - ((d + i) % 1000) * 86400000;
|
||||
row.youngestMedia = now - ((d + i) % 100) * 86400000;
|
||||
row.coverMediaId = ((d + i) % 3 === 0) ? ((d + i) % 1000) : null;
|
||||
row.updatedAt = now;
|
||||
batch.push(row);
|
||||
}
|
||||
}
|
||||
await upsertBatch(dirARepo, batch, ['scopeKey', 'directoryId']);
|
||||
});
|
||||
|
||||
await time('A.upsert ScopedPersonA', async () => {
|
||||
const batch: ScopedPersonA[] = [];
|
||||
const now = Date.now();
|
||||
for (let i = 0; i < N; i++) {
|
||||
for (let p = 1; p <= P; p++) {
|
||||
const row = new ScopedPersonA();
|
||||
row.scopeKey = scopeKeys[i];
|
||||
row.personId = p;
|
||||
row.count = (p * 11 + i) % 30;
|
||||
row.sampleRegionId = ((p + i) % 4 === 0) ? ((p + i) % 5000) : null;
|
||||
row.updatedAt = now;
|
||||
batch.push(row);
|
||||
}
|
||||
}
|
||||
await upsertBatch(personARepo, batch, ['scopeKey', 'personId']);
|
||||
});
|
||||
|
||||
// Create data for Variant B
|
||||
await time('B.upsert ScopedDirB', async () => {
|
||||
const batch: ScopedDirB[] = [];
|
||||
const now = Date.now();
|
||||
for (let i = 0; i < N; i++) {
|
||||
const scopeId = scopeIdByKey.get(scopeKeys[i])!;
|
||||
for (let d = 1; d <= D; d++) {
|
||||
const row = new ScopedDirB();
|
||||
row.scope = { id: scopeId } as any;
|
||||
row.directoryId = d;
|
||||
row.mediaCount = (d * 7 + i) % 50;
|
||||
row.oldestMedia = now - ((d + i) % 1000) * 86400000;
|
||||
row.youngestMedia = now - ((d + i) % 100) * 86400000;
|
||||
row.coverMediaId = ((d + i) % 3 === 0) ? ((d + i) % 1000) : null;
|
||||
row.updatedAt = now;
|
||||
batch.push(row);
|
||||
}
|
||||
}
|
||||
await upsertBatch(dirBRepo, batch, ['scope', 'directoryId']);
|
||||
});
|
||||
|
||||
await time('B.upsert ScopedPersonB', async () => {
|
||||
const batch: ScopedPersonB[] = [];
|
||||
const now = Date.now();
|
||||
for (let i = 0; i < N; i++) {
|
||||
const scopeId = scopeIdByKey.get(scopeKeys[i])!;
|
||||
for (let p = 1; p <= P; p++) {
|
||||
const row = new ScopedPersonB();
|
||||
row.scope = { id: scopeId } as any;
|
||||
row.personId = p;
|
||||
row.count = (p * 11 + i) % 30;
|
||||
row.sampleRegionId = ((p + i) % 4 === 0) ? ((p + i) % 5000) : null;
|
||||
row.updatedAt = now;
|
||||
batch.push(row);
|
||||
}
|
||||
}
|
||||
await upsertBatch(personBRepo, batch, ['scope', 'personId']);
|
||||
});
|
||||
|
||||
// Lookup tests
|
||||
await time('A.lookup ScopedDirA', async () => {
|
||||
for (let i = 0; i < lookups; i++) {
|
||||
const k = scopeKeys[randInt(N)];
|
||||
const d = 1 + randInt(D);
|
||||
await dirARepo.findOne({ where: { scopeKey: k, directoryId: d } });
|
||||
}
|
||||
});
|
||||
|
||||
await time('B.lookup ScopedDirB', async () => {
|
||||
for (let i = 0; i < lookups; i++) {
|
||||
const k = scopeKeys[randInt(N)];
|
||||
const scopeId = scopeIdByKey.get(k)!;
|
||||
const d = 1 + randInt(D);
|
||||
await dirBRepo.createQueryBuilder('b').where('b.scopeId = :sid AND b.directoryId = :d', { sid: scopeId, d }).getOne();
|
||||
}
|
||||
});
|
||||
|
||||
// Cascade delete tests (delete half of scopes)
|
||||
const half = Math.floor(N / 2);
|
||||
const deleteKeys = scopeKeys.slice(0, half);
|
||||
|
||||
const sizeBefore = fs.statSync(dbPath).size;
|
||||
|
||||
await time('A.delete scope rows by scopeKey (2 tables)', async () => {
|
||||
// delete in both A tables
|
||||
await dirARepo.createQueryBuilder().delete().where('scopeKey IN (:...keys)', { keys: deleteKeys }).execute();
|
||||
await personARepo.createQueryBuilder().delete().where('scopeKey IN (:...keys)', { keys: deleteKeys }).execute();
|
||||
});
|
||||
|
||||
await time('B.delete ScopeKey rows (cascade)', async () => {
|
||||
await scopeRepoB.createQueryBuilder().delete().where('key IN (:...keys)', { keys: deleteKeys }).execute();
|
||||
});
|
||||
|
||||
const sizeAfter = fs.statSync(dbPath).size;
|
||||
|
||||
console.log('\nSQLite DB file size (bytes): before:', sizeBefore, 'after:', sizeAfter, 'delta:', sizeAfter - sizeBefore);
|
||||
|
||||
await ds.destroy();
|
||||
|
||||
console.log('\nDone. You can tweak parameters via env vars: BENCH_SCOPES, BENCH_DIRS, BENCH_PERSONS, BENCH_LOOKUPS');
|
||||
})();
|
||||
@@ -1,27 +1,33 @@
|
||||
import * as fs from 'fs';
|
||||
|
||||
import { Config } from '../../../common/config/private/Config';
|
||||
import { FaceRegion, PhotoMetadata } from '../../../common/entities/PhotoDTO';
|
||||
import { VideoMetadata } from '../../../common/entities/VideoDTO';
|
||||
import { RatingTypes } from '../../../common/entities/MediaDTO';
|
||||
import { Logger } from '../../Logger';
|
||||
import {Config} from '../../../common/config/private/Config';
|
||||
import {FaceRegion, PhotoMetadata} from '../../../common/entities/PhotoDTO';
|
||||
import {VideoMetadata} from '../../../common/entities/VideoDTO';
|
||||
import {RatingTypes} from '../../../common/entities/MediaDTO';
|
||||
import {Logger} from '../../Logger';
|
||||
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
|
||||
// @ts-ignore
|
||||
import * as exifr from 'exifr';
|
||||
import { FfprobeData } from 'fluent-ffmpeg';
|
||||
import { FileHandle } from 'fs/promises';
|
||||
import {FfprobeData} from 'fluent-ffmpeg';
|
||||
import * as util from 'node:util';
|
||||
import * as path from 'path';
|
||||
import { Utils } from '../../../common/Utils';
|
||||
import { FFmpegFactory } from '../FFmpegFactory';
|
||||
import { ExtensionDecorator } from '../extension/ExtensionDecorator';
|
||||
import { DateTags } from './MetadataCreationDate';
|
||||
const { imageSizeFromFile } = require('image-size/fromFile')
|
||||
import {Utils} from '../../../common/Utils';
|
||||
import {FFmpegFactory} from '../FFmpegFactory';
|
||||
import {ExtensionDecorator} from '../extension/ExtensionDecorator';
|
||||
import {DateTags} from './MetadataCreationDate';
|
||||
|
||||
const {imageSizeFromFile} = require('image-size/fromFile');
|
||||
const LOG_TAG = '[MetadataLoader]';
|
||||
const ffmpeg = FFmpegFactory.get();
|
||||
|
||||
export class MetadataLoader {
|
||||
|
||||
private static readonly EMPTY_METADATA: PhotoMetadata = {
|
||||
size: {width: 0, height: 0},
|
||||
creationDate: 0,
|
||||
fileSize: 0,
|
||||
};
|
||||
|
||||
@ExtensionDecorator(e => e.gallery.MetadataLoader.loadVideoMetadata)
|
||||
public static async loadVideoMetadata(fullPath: string): Promise<VideoMetadata> {
|
||||
const metadata: VideoMetadata = {
|
||||
@@ -161,17 +167,10 @@ export class MetadataLoader {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
private static readonly EMPTY_METADATA: PhotoMetadata = {
|
||||
size: { width: 0, height: 0 },
|
||||
creationDate: 0,
|
||||
fileSize: 0,
|
||||
};
|
||||
|
||||
@ExtensionDecorator(e => e.gallery.MetadataLoader.loadPhotoMetadata)
|
||||
public static async loadPhotoMetadata(fullPath: string): Promise<PhotoMetadata> {
|
||||
let fileHandle: FileHandle;
|
||||
const metadata: PhotoMetadata = {
|
||||
size: { width: 0, height: 0 },
|
||||
size: {width: 0, height: 0},
|
||||
creationDate: 0,
|
||||
fileSize: 0,
|
||||
};
|
||||
@@ -189,12 +188,9 @@ export class MetadataLoader {
|
||||
mergeOutput: false //don't merge output, because things like Microsoft Rating (percent) and xmp.rating will be merged
|
||||
};
|
||||
try {
|
||||
let bufferSize = Config.Media.photoMetadataSize;
|
||||
try {
|
||||
const stat = fs.statSync(fullPath);
|
||||
metadata.fileSize = stat.size;
|
||||
//No reason to make the buffer larger than the actual file
|
||||
bufferSize = Math.min(Config.Media.photoMetadataSize, metadata.fileSize);
|
||||
metadata.creationDate = stat.mtime.getTime();
|
||||
} catch (err) {
|
||||
// ignoring errors
|
||||
@@ -202,10 +198,10 @@ export class MetadataLoader {
|
||||
try {
|
||||
//read the actual image size, don't rely on tags for this
|
||||
const info = await imageSizeFromFile(fullPath);
|
||||
metadata.size = { width: info.width, height: info.height };
|
||||
metadata.size = {width: info.width, height: info.height};
|
||||
} catch (e) {
|
||||
//in case of failure, set dimensions to 0 so they may be read via tags
|
||||
metadata.size = { width: 0, height: 0 };
|
||||
metadata.size = {width: 0, height: 0};
|
||||
} finally {
|
||||
if (isNaN(metadata.size.width) || metadata.size.width == null) {
|
||||
metadata.size.width = 0;
|
||||
@@ -215,21 +211,9 @@ export class MetadataLoader {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
const data = Buffer.allocUnsafe(bufferSize);
|
||||
fileHandle = await fs.promises.open(fullPath, 'r');
|
||||
try {
|
||||
await fileHandle.read(data, 0, bufferSize, 0);
|
||||
} catch (err) {
|
||||
Logger.error(LOG_TAG, 'Error during reading photo: ' + fullPath);
|
||||
console.error(err);
|
||||
return MetadataLoader.EMPTY_METADATA;
|
||||
} finally {
|
||||
await fileHandle.close();
|
||||
}
|
||||
try {
|
||||
try {
|
||||
const exif = await exifr.parse(data, exifrOptions);
|
||||
const exif = await exifr.parse(fullPath, exifrOptions);
|
||||
MetadataLoader.mapMetadata(metadata, exif);
|
||||
} catch (err) {
|
||||
// ignoring errors
|
||||
@@ -297,6 +281,7 @@ export class MetadataLoader {
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private static getOrientation(exif: any): number {
|
||||
let orientation = 1; //Orientation 1 is normal
|
||||
if (exif.ifd0?.Orientation != undefined) {
|
||||
@@ -360,7 +345,7 @@ export class MetadataLoader {
|
||||
}
|
||||
|
||||
private static mapCaption(metadata: PhotoMetadata, exif: any) {
|
||||
metadata.caption = exif.dc?.description?.value || Utils.asciiToUTF8(exif.iptc?.Caption) || metadata.caption || exif.ifd0?.ImageDescription || exif.exif?.UserComment?.value || exif.Iptc4xmpCore?.ExtDescrAccessibility?.value ||exif.acdsee?.notes;
|
||||
metadata.caption = exif.dc?.description?.value || Utils.asciiToUTF8(exif.iptc?.Caption) || metadata.caption || exif.ifd0?.ImageDescription || exif.exif?.UserComment?.value || exif.Iptc4xmpCore?.ExtDescrAccessibility?.value || exif.acdsee?.notes;
|
||||
}
|
||||
|
||||
private static mapTimestampAndOffset(metadata: PhotoMetadata, exif: any) {
|
||||
@@ -376,7 +361,7 @@ export class MetadataLoader {
|
||||
}
|
||||
if (!offset) { //still no offset? let's look for a timestamp with offset in the rest of the DateTags list
|
||||
const [tsonly, tsoffset] = Utils.splitTimestampAndOffset(ts);
|
||||
for (let j = i+1; j < DateTags.length; j++) {
|
||||
for (let j = i + 1; j < DateTags.length; j++) {
|
||||
const [exts, exOffset] = extractTSAndOffset(DateTags[j][0], DateTags[j][1], DateTags[j][2]);
|
||||
if (exts && exOffset && Math.abs(Utils.timestampToMS(tsonly, null) - Utils.timestampToMS(exts, null)) < 30000) {
|
||||
//if there is an offset and the found timestamp is within 30 seconds of the extra timestamp, we will use the offset from the found timestamp
|
||||
@@ -473,10 +458,10 @@ export class MetadataLoader {
|
||||
metadata.positionData.GPSData.latitude = Utils.isFloat32(exif.gps?.latitude) ? exif.gps.latitude : Utils.xmpExifGpsCoordinateToDecimalDegrees(exif.exif.GPSLatitude);
|
||||
|
||||
if (metadata.positionData.GPSData.longitude !== undefined) {
|
||||
metadata.positionData.GPSData.longitude = parseFloat(metadata.positionData.GPSData.longitude.toFixed(6))
|
||||
metadata.positionData.GPSData.longitude = parseFloat(metadata.positionData.GPSData.longitude.toFixed(6));
|
||||
}
|
||||
if (metadata.positionData.GPSData.latitude !== undefined) {
|
||||
metadata.positionData.GPSData.latitude = parseFloat(metadata.positionData.GPSData.latitude.toFixed(6))
|
||||
metadata.positionData.GPSData.latitude = parseFloat(metadata.positionData.GPSData.latitude.toFixed(6));
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
@@ -524,10 +509,10 @@ export class MetadataLoader {
|
||||
|
||||
private static mapFaces(metadata: PhotoMetadata, exif: any, orientation: number) {
|
||||
//xmp."mwg-rs" section
|
||||
if (exif["mwg-rs"] &&
|
||||
exif["mwg-rs"].Regions) {
|
||||
if (exif['mwg-rs'] &&
|
||||
exif['mwg-rs'].Regions) {
|
||||
const faces: FaceRegion[] = [];
|
||||
const regionListVal = Array.isArray(exif["mwg-rs"].Regions.RegionList) ? exif["mwg-rs"].Regions.RegionList : [exif["mwg-rs"].Regions.RegionList];
|
||||
const regionListVal = Array.isArray(exif['mwg-rs'].Regions.RegionList) ? exif['mwg-rs'].Regions.RegionList : [exif['mwg-rs'].Regions.RegionList];
|
||||
if (regionListVal) {
|
||||
for (const regionRoot of regionListVal) {
|
||||
let type;
|
||||
@@ -613,7 +598,7 @@ export class MetadataLoader {
|
||||
box.top = Math.round(Math.max(0, box.top - box.height / 2));
|
||||
|
||||
|
||||
faces.push({ name, box });
|
||||
faces.push({name, box});
|
||||
}
|
||||
}
|
||||
if (faces.length > 0) {
|
||||
|
||||
@@ -904,19 +904,6 @@ export class ServerMediaConfig extends ClientMediaConfig {
|
||||
})
|
||||
tempFolder: string = 'demo/tmp';
|
||||
|
||||
@ConfigProperty({
|
||||
type: 'unsignedInt',
|
||||
tags: {
|
||||
name: $localize`Metadata read buffer`,
|
||||
priority: ConfigPriority.underTheHood,
|
||||
uiResetNeeded: {db: true, server: true},
|
||||
githubIssue: 398,
|
||||
unit: 'bytes'
|
||||
} as TAGS,
|
||||
description: $localize`Only this many bites will be loaded when scanning photo/video for metadata. Increase this number if your photos shows up as square.`,
|
||||
})
|
||||
photoMetadataSize: number = 512 * 1024; // only this many bites will be loaded when scanning photo for metadata
|
||||
|
||||
@ConfigProperty({
|
||||
tags: {
|
||||
name: $localize`Video`,
|
||||
|
||||
@@ -8,6 +8,9 @@ const dir = process.argv[2];
|
||||
if(dir === 'test'){
|
||||
throw new Error('test folder is not allowed to be deleted');
|
||||
}
|
||||
if(dir === '--exclude'){
|
||||
throw new Error('--exclude folder is not allowed to be deleted/created');
|
||||
}
|
||||
if (fs.existsSync(dir)) {
|
||||
console.log('deleting folder:' + dir);
|
||||
fs.rmSync(dir, {recursive: true});
|
||||
|
||||
Reference in New Issue
Block a user