self-hosted/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2024-11-24 08:52:28 +02:00
Code Issues Releases Activity

feat(server): Automatic watching of library folders (#6192)

Browse Source 
* feat: initial watch support

* allow offline files

* chore: ignore query errors when resetting e2e db

* revert db query

* add savepoint

* guard the user query

* chore: openapi and db migration

* wip

* support multiple libraries

* fix tests

* wip

* can now cleanup chokidar watchers

* fix unit tests

* add library watch queue

* add missing init from merge

* wip

* can now filter file extensions

* remove watch api from non job client

* Fix e2e test

* watch library with updated import path and exclusion pattern

* add library watch frontend ui

* case sensitive watching extensions

* can auto watch libraries

* move watcher e2e tests to separate file

* don't watch libraries from a queue

* use event emitters

* shorten e2e test timeout

* refactor chokidar code to filesystem provider

* expose chokidar parameters to config file

* fix storage mock

* set default config for library watching

* add fs provider mocks

* cleanup

* add more unit tests for watcher

* chore: fix format + sql

* add more tests

* move unwatch feature back to library service

* add file event unit tests

* chore: formatting

* add documentation

* fix e2e tests

* chore: fix e2e tests

* fix library updating

* test cleanup

* fix typo

* cleanup

* fixing as per pr comments

* reduce library watch config file

* update storage config and mocks

* move negative event tests to unit tests

* fix library watcher e2e

* make watch configuration global

* remove the feature flag

* refactor watcher teardown

* fix microservices init

* centralize asset scan job queue

* improve docs

* add more tests

* chore: open api

* initialize app service

* fix docs

* fix library watch feature flag

* Update docs/docs/features/libraries.md

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

* fix: import right app service

* don't be truthy

* fix test speling

* stricter library update tests

* move fs watcher mock to external file

* subscribe to config changes

* docker does not need polling

* make library watch() private

* feat: add configuration ui

---------

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
This commit is contained in:
Jonathan Jogenfors 2024-01-31 09:15:54 +01:00 committed by GitHub
parent 4079e92bbf
commit 068e703e88
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
48 changed files with 1400 additions and 109 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/docs/features/libraries.md
View File

@ -72,7 +72,7 @@ For security purposes, each Immich user is disallowed to add external files by d
With the `external path` set, a user is restricted to accessing external files to files or directories within that path. The Immich admin should still be careful not set the external path too generously. For example, `user1` wants to read their photos in to `/home/user1`. A lazy admin sets that user's external path to `/home/` since it "gets the job done". However, that user will then be able to read all photos in `/home/user2/private-photos`, too! Please set the external path as specific as possible. If multiple folders must be added, do this using the docker volume mount feature described below.
### Exclusion Patterns and Scan Settings
### Exclusion Patterns
By default, all files in the import paths will be added to the library. If there are files that should not be added, exclusion patterns can be used to exclude them. Exclusion patterns are glob patterns are matched against the full file path. If a file matches an exclusion pattern, it will not be added to the library. Exclusion patterns can be added in the Scan Settings page for each library. Under the hood, Immich uses the [glob](https://www.npmjs.com/package/glob) package to match patterns, so please refer to [their documentation](https://github.com/isaacs/node-glob#glob-primer) to see what patterns are supported.
@ -83,6 +83,15 @@ Some basic examples:
- `**/Raw/**` will exclude all files in any directory named `Raw`
- `**/*.{tif,jpg}` will exclude all files with the extension `.tif` or `.jpg`
### Automatic watching (EXPERIMENTAL)
This feature - currently hidden in the config file - is considered experimental and for advanced users only. If enabled, it will allow automatic watching of the filesystem which means new assets are automatically imported to Immich without needing to rescan. Deleted assets are, as always, marked as offline and can be removed with the "Remove offline files" button.
If your photos are on a network drive you will likely have to enable filesystem polling. The performance hit for polling large libraries is currently unknown, feel free to test this feature and report back. In addition to the boolean feature flag, the configuration file allows customization of the following parameters, please see the [chokidar documentation](https://github.com/paulmillr/chokidar?tab=readme-ov-file#performance) for reference.
- `usePolling` (default: `false`).
- `interval`. (default: 10000). When using polling, this is how often (in milliseconds) the filesystem is polled.
### Nightly job
There is an automatic job that's run once a day and refreshes all modified files in all libraries as well as cleans up any libraries stuck in deletion.

5
docs/docs/install/config-file.md
View File

@ -129,6 +129,11 @@ The default configuration looks like this:
"scan": {
"enabled": true,
"cronExpression": "0 0 * * *"
},
"watch": {
"enabled": false,
"usePolling": false,
"interval": 10000
}
}
}

10
docs/src/pages/milestones.tsx
View File

@ -1,4 +1,5 @@
import {
mdiEyeRefreshOutline,
mdiAccountGroup,
mdiAndroid,
mdiAppleIos,
@ -54,6 +55,15 @@ import React from 'react';
import Timeline, { DateType, Item } from '../components/timeline';
const items: Item[] = [
{
icon: mdiEyeRefreshOutline,
description: 'Automatically import files in external libraries when the operating system detects changes.',
title: 'Library watching',
release: 'v1.94.0',
tag: 'v1.94.0',
date: new Date(2024, 1, 31),
dateType: DateType.RELEASE,
},
{
icon: mdiMatrix,
description: 'Moved the search from typesense to pgvecto.rs',

3
mobile/openapi/.openapi-generator/FILES generated
View File

@ -143,6 +143,7 @@ doc/SystemConfigFFmpegDto.md
doc/SystemConfigJobDto.md
doc/SystemConfigLibraryDto.md
doc/SystemConfigLibraryScanDto.md
doc/SystemConfigLibraryWatchDto.md
doc/SystemConfigLoggingDto.md
doc/SystemConfigMachineLearningDto.md
doc/SystemConfigMapDto.md
@ -333,6 +334,7 @@ lib/model/system_config_f_fmpeg_dto.dart
lib/model/system_config_job_dto.dart
lib/model/system_config_library_dto.dart
lib/model/system_config_library_scan_dto.dart
lib/model/system_config_library_watch_dto.dart
lib/model/system_config_logging_dto.dart
lib/model/system_config_machine_learning_dto.dart
lib/model/system_config_map_dto.dart
@ -508,6 +510,7 @@ test/system_config_f_fmpeg_dto_test.dart
test/system_config_job_dto_test.dart
test/system_config_library_dto_test.dart
test/system_config_library_scan_dto_test.dart
test/system_config_library_watch_dto_test.dart
test/system_config_logging_dto_test.dart
test/system_config_machine_learning_dto_test.dart
test/system_config_map_dto_test.dart

BIN
mobile/openapi/README.md generated
View File

Binary file not shown.

BIN
mobile/openapi/doc/CreateLibraryDto.md generated
View File

Binary file not shown.

BIN
mobile/openapi/doc/SystemConfigLibraryDto.md generated
View File

Binary file not shown.

BIN
mobile/openapi/doc/SystemConfigLibraryWatchDto.md generated Normal file
View File

Binary file not shown.

BIN
mobile/openapi/lib/api.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/lib/api_client.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/lib/model/create_library_dto.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/lib/model/system_config_library_dto.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/lib/model/system_config_library_watch_dto.dart generated Normal file
View File

Binary file not shown.

BIN
mobile/openapi/test/create_library_dto_test.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/test/system_config_library_dto_test.dart generated
View File

Binary file not shown.

BIN
mobile/openapi/test/system_config_library_watch_dto_test.dart generated Normal file
View File

Binary file not shown.

28
open-api/immich-openapi-specs.json
View File

@ -7601,6 +7601,9 @@
"isVisible": {
"type": "boolean"
},
"isWatched": {
"type": "boolean"
},
"name": {
"type": "string"
},
@ -9531,10 +9534,14 @@
"properties": {
"scan": {
"$ref": "#/components/schemas/SystemConfigLibraryScanDto"
},
"watch": {
"$ref": "#/components/schemas/SystemConfigLibraryWatchDto"
}
},
"required": [
"scan"
"scan",
"watch"
],
"type": "object"
},
@ -9553,6 +9560,25 @@
],
"type": "object"
},
"SystemConfigLibraryWatchDto": {
"properties": {
"enabled": {
"type": "boolean"
},
"interval": {
"type": "integer"
},
"usePolling": {
"type": "boolean"
}
},
"required": [
"enabled",
"interval",
"usePolling"
],
"type": "object"
},
"SystemConfigLoggingDto": {
"properties": {
"enabled": {

37
open-api/typescript-sdk/client/api.ts generated
View File

@ -1387,6 +1387,12 @@ export interface CreateLibraryDto {
* @memberof CreateLibraryDto
*/
'isVisible'?: boolean;
/**
*
* @type {boolean}
* @memberof CreateLibraryDto
*/
'isWatched'?: boolean;
/**
*
* @type {string}
@ -3860,6 +3866,12 @@ export interface SystemConfigLibraryDto {
* @memberof SystemConfigLibraryDto
*/
'scan': SystemConfigLibraryScanDto;
/**
*
* @type {SystemConfigLibraryWatchDto}
* @memberof SystemConfigLibraryDto
*/
'watch': SystemConfigLibraryWatchDto;
}
/**
*
@ -3880,6 +3892,31 @@ export interface SystemConfigLibraryScanDto {
*/
'enabled': boolean;
}
/**
*
* @export
* @interface SystemConfigLibraryWatchDto
*/
export interface SystemConfigLibraryWatchDto {
/**
*
* @type {boolean}
* @memberof SystemConfigLibraryWatchDto
*/
'enabled': boolean;
/**
*
* @type {number}
* @memberof SystemConfigLibraryWatchDto
*/
'interval': number;
/**
*
* @type {boolean}
* @memberof SystemConfigLibraryWatchDto
*/
'usePolling': boolean;
}
/**
*
* @export

1
server/e2e/api/jest-e2e.json
View File

@ -5,7 +5,6 @@
"globalSetup": "<rootDir>/e2e/api/setup.ts",
"testEnvironment": "node",
"testMatch": ["**/e2e/api/specs/*.e2e-spec.[tj]s"],
"testTimeout": 60000,
"transform": {
"^.+\\.(t|j)s$": "ts-jest"
},

16
server/e2e/client/library-api.ts
View File

@ -1,4 +1,10 @@
import { CreateLibraryDto, LibraryResponseDto, LibraryStatsResponseDto, ScanLibraryDto } from '@app/domain';
import {
CreateLibraryDto,
LibraryResponseDto,
LibraryStatsResponseDto,
ScanLibraryDto,
UpdateLibraryDto,
} from '@app/domain';
import request from 'supertest';
export const libraryApi = {
@ -44,4 +50,12 @@ export const libraryApi = {
expect(status).toBe(200);
return body;
},
update: async (server: any, accessToken: string, id: string, data: UpdateLibraryDto) => {
const { body, status } = await request(server)
.put(`/library/${id}`)
.set('Authorization', `Bearer ${accessToken}`)
.send(data);
expect(status).toBe(200);
return body as LibraryResponseDto;
},
};

17
server/e2e/jobs/config/library-watcher-e2e-config.json Normal file
View File

@ -0,0 +1,17 @@
{
"reverseGeocoding": {
"enabled": false
},
"machineLearning": {
"enabled": false
},
"logging": {
"enabled": false,
"level": "debug"
},
"library": {
"watch": {
"enabled": true
}
}
}

3
server/e2e/jobs/immich-e2e-config.json
View File

@ -6,6 +6,7 @@
"enabled": false
},
"logging": {
"enabled": false
"enabled": false,
"level": "debug"
}
}

2
server/e2e/jobs/jest-e2e.json
View File

@ -5,7 +5,7 @@
"globalSetup": "<rootDir>/e2e/jobs/setup.ts",
"testEnvironment": "node",
"testMatch": ["**/e2e/jobs/specs/*.e2e-spec.[tj]s"],
"testTimeout": 60000,
"testTimeout": 10000,
"transform": {
"^.+\\.(t|j)s$": "ts-jest"
},

235
server/e2e/jobs/specs/library-watcher.e2e-spec.ts Normal file
View File

@ -0,0 +1,235 @@
import { LibraryResponseDto, LibraryService, LoginResponseDto } from '@app/domain';
import { AssetType, LibraryType } from '@app/infra/entities';
import * as fs from 'fs/promises';
import { api } from '../../client';
import path from 'path';
import {
IMMICH_TEST_ASSET_PATH,
IMMICH_TEST_ASSET_TEMP_PATH,
restoreTempFolder,
testApp,
waitForEvent,
} from '../../../src/test-utils/utils';
describe(`Library watcher (e2e)`, () => {
let server: any;
let admin: LoginResponseDto;
let libraryService: LibraryService;
const configFilePath = process.env.IMMICH_CONFIG_FILE;
beforeAll(async () => {
process.env.IMMICH_CONFIG_FILE = path.normalize(`${__dirname}/../config/library-watcher-e2e-config.json`);
server = (await testApp.create()).getHttpServer();
libraryService = testApp.get(LibraryService);
});
beforeEach(async () => {
await testApp.reset();
await restoreTempFolder();
await api.authApi.adminSignUp(server);
admin = await api.authApi.adminLogin(server);
await api.userApi.setExternalPath(server, admin.accessToken, admin.userId, '/');
});
afterEach(async () => {
await libraryService.unwatchAll();
});
afterAll(async () => {
await testApp.teardown();
await restoreTempFolder();
process.env.IMMICH_CONFIG_FILE = configFilePath;
});
describe('Event handling', () => {
let library: LibraryResponseDto;
describe('Single import path', () => {
beforeEach(async () => {
library = await api.libraryApi.create(server, admin.accessToken, {
type: LibraryType.EXTERNAL,
importPaths: [`${IMMICH_TEST_ASSET_TEMP_PATH}`],
});
});
it('should import a new file', async () => {
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file.jpg`,
);
await waitForEvent(libraryService, 'add');
const afterAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(afterAssets.length).toEqual(1);
});
it('should import new files with case insensitive extensions', async () => {
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file2.JPG`,
);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file3.Jpg`,
);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file4.jpG`,
);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file5.jPg`,
);
await waitForEvent(libraryService, 'add');
await waitForEvent(libraryService, 'add');
await waitForEvent(libraryService, 'add');
await waitForEvent(libraryService, 'add');
const afterAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(afterAssets.length).toEqual(4);
});
it('should update a changed file', async () => {
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file.jpg`,
);
await waitForEvent(libraryService, 'add');
const originalAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(originalAssets.length).toEqual(1);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/prairie_falcon.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/file.jpg`,
);
await waitForEvent(libraryService, 'change');
const afterAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(afterAssets).toEqual([
expect.objectContaining({
// Make sure we keep the original asset id
id: originalAssets[0].id,
type: AssetType.IMAGE,
exifInfo: expect.objectContaining({
make: 'Canon',
model: 'Canon EOS R5',
exifImageWidth: 800,
exifImageHeight: 533,
exposureTime: '1/4000',
}),
}),
]);
});
});
describe('Multiple import paths', () => {
beforeEach(async () => {
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1`, { recursive: true });
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir2`, { recursive: true });
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir3`, { recursive: true });
library = await api.libraryApi.create(server, admin.accessToken, {
type: LibraryType.EXTERNAL,
importPaths: [
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir2`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir3`,
],
});
});
it('should add new files in multiple import paths', async () => {
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/el_torcal_rocks.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1/file2.jpg`,
);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/polemonium_reptans.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir2/file3.jpg`,
);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/tanners_ridge.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir3/file4.jpg`,
);
await waitForEvent(libraryService, 'add');
await waitForEvent(libraryService, 'add');
await waitForEvent(libraryService, 'add');
const assets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(assets.length).toEqual(3);
});
it('should offline a removed file', async () => {
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/polemonium_reptans.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1/file.jpg`,
);
await waitForEvent(libraryService, 'add');
const addedAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(addedAssets.length).toEqual(1);
await fs.unlink(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1/file.jpg`);
await waitForEvent(libraryService, 'unlink');
const afterAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(afterAssets[0].isOffline).toEqual(true);
});
});
});
describe('Configuration', () => {
let library: LibraryResponseDto;
beforeEach(async () => {
library = await api.libraryApi.create(server, admin.accessToken, {
type: LibraryType.EXTERNAL,
importPaths: [
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir2`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir3`,
],
});
await api.userApi.setExternalPath(server, admin.accessToken, admin.userId, '/');
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir1`, { recursive: true });
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir2`, { recursive: true });
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir3`, { recursive: true });
});
it('should use an updated import paths', async () => {
await fs.mkdir(`${IMMICH_TEST_ASSET_TEMP_PATH}/dir4`, { recursive: true });
await api.libraryApi.setImportPaths(server, admin.accessToken, library.id, [
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir4`,
]);
await fs.copyFile(
`${IMMICH_TEST_ASSET_PATH}/albums/nature/polemonium_reptans.jpg`,
`${IMMICH_TEST_ASSET_TEMP_PATH}/dir4/file.jpg`,
);
await waitForEvent(libraryService, 'add');
const afterAssets = await api.assetApi.getAllAssets(server, admin.accessToken);
expect(afterAssets.length).toEqual(1);
});
});
});

10
server/e2e/jobs/specs/library.e2e-spec.ts
View File

@ -21,11 +21,6 @@ describe(`${LibraryController.name} (e2e)`, () => {
server = (await testApp.create()).getHttpServer();
});
afterAll(async () => {
await testApp.teardown();
await restoreTempFolder();
});
beforeEach(async () => {
await testApp.reset();
await restoreTempFolder();
@ -33,6 +28,11 @@ describe(`${LibraryController.name} (e2e)`, () => {
admin = await api.authApi.adminLogin(server);
});
afterAll(async () => {
await testApp.teardown();
await restoreTempFolder();
});
describe('DELETE /library/:id', () => {
it('should delete an external library with assets', async () => {
const library = await api.libraryApi.create(server, admin.accessToken, {

7
server/e2e/jobs/specs/metadata.e2e-spec.ts
View File

@ -1,6 +1,5 @@
import { AssetResponseDto, LoginResponseDto } from '@app/domain';
import { AssetController } from '@app/immich';
import { INestApplication } from '@nestjs/common';
import { exiftool } from 'exiftool-vendored';
import { readFile, writeFile } from 'fs/promises';
import {
@ -13,17 +12,15 @@ import {
import { api } from '../../client';
describe(`${AssetController.name} (e2e)`, () => {
let app: INestApplication;
let server: any;
let admin: LoginResponseDto;
beforeAll(async () => {
app = await testApp.create();
server = app.getHttpServer();
server = (await testApp.create()).getHttpServer();
});
beforeEach(async () => {
await db.reset();
await testApp.reset();
await restoreTempFolder();
await api.authApi.adminSignUp(server);
admin = await api.authApi.adminLogin(server);

129
server/package-lock.json generated
View File

@ -22,6 +22,7 @@
"@nestjs/typeorm": "^10.0.0",
"@nestjs/websockets": "^10.2.2",
"@socket.io/postgres-adapter": "^0.3.1",
"@types/picomatch": "^2.3.3",
"archiver": "^6.0.0",
"async-lock": "^1.4.0",
"axios": "^1.5.0",
@ -45,6 +46,7 @@
"node-addon-api": "^7.0.0",
"openid-client": "^5.4.3",
"pg": "^8.11.3",
"picomatch": "^3.0.1",
"reflect-metadata": "^0.1.13",
"rxjs": "^7.8.1",
"sanitize-filename": "^1.6.3",
@ -76,6 +78,7 @@
"@types/ua-parser-js": "^0.7.36",
"@typescript-eslint/eslint-plugin": "^6.4.1",
"@typescript-eslint/parser": "^6.4.1",
"chokidar": "^3.5.3",
"dotenv": "^16.3.1",
"eslint": "^8.48.0",
"eslint-config-prettier": "^9.0.0",
@ -147,18 +150,6 @@
}
}
},
"node_modules/@angular-devkit/core/node_modules/picomatch": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-3.0.1.tgz",
"integrity": "sha512-I3EurrIQMlRc9IaAZnqRR044Phh2DXY+55o7uJ0V+hYZAcQYSuFWsc9q5PvyDHUSCe1Qxn/iBz+78s86zWnGag==",
"dev": true,
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/@angular-devkit/schematics": {
"version": "17.0.9",
"resolved": "https://registry.npmjs.org/@angular-devkit/schematics/-/schematics-17.0.9.tgz",
@ -3304,6 +3295,11 @@
"node": ">=12"
}
},
"node_modules/@types/picomatch": {
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/@types/picomatch/-/picomatch-2.3.3.tgz",
"integrity": "sha512-Yll76ZHikRFCyz/pffKGjrCwe/le2CDwOP5F210KQo27kpRE46U2rDnzikNlVn6/ezH3Mhn46bJMTfeVTtcYMg=="
},
"node_modules/@types/qs": {
"version": "6.9.8",
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.9.8.tgz",
@ -3994,6 +3990,18 @@
"node": ">= 8"
}
},
"node_modules/anymatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true,
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/app-root-path": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/app-root-path/-/app-root-path-3.1.0.tgz",
@ -7924,6 +7932,18 @@
"node": "^14.15.0 || ^16.10.0 || >=18.0.0"
}
},
"node_modules/jest-util/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true,
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/jest-validate": {
"version": "29.7.0",
"resolved": "https://registry.npmjs.org/jest-validate/-/jest-validate-29.7.0.tgz",
@ -8417,6 +8437,18 @@
"node": ">=8.6"
}
},
"node_modules/micromatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true,
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/mime": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
@ -9287,12 +9319,11 @@
"dev": true
},
"node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true,
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-3.0.1.tgz",
"integrity": "sha512-I3EurrIQMlRc9IaAZnqRR044Phh2DXY+55o7uJ0V+hYZAcQYSuFWsc9q5PvyDHUSCe1Qxn/iBz+78s86zWnGag==",
"engines": {
"node": ">=8.6"
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
@ -9768,6 +9799,18 @@
"node": ">=8.10.0"
}
},
"node_modules/readdirp/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true,
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/rechoir": {
"version": "0.6.2",
"resolved": "https://registry.npmjs.org/rechoir/-/rechoir-0.6.2.tgz",
@ -12248,14 +12291,6 @@
"picomatch": "3.0.1",
"rxjs": "7.8.1",
"source-map": "0.7.4"
},
"dependencies": {
"picomatch": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-3.0.1.tgz",
"integrity": "sha512-I3EurrIQMlRc9IaAZnqRR044Phh2DXY+55o7uJ0V+hYZAcQYSuFWsc9q5PvyDHUSCe1Qxn/iBz+78s86zWnGag==",
"dev": true
}
}
},
"@angular-devkit/schematics": {
@ -14468,6 +14503,11 @@
}
}
},
"@types/picomatch": {
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/@types/picomatch/-/picomatch-2.3.3.tgz",
"integrity": "sha512-Yll76ZHikRFCyz/pffKGjrCwe/le2CDwOP5F210KQo27kpRE46U2rDnzikNlVn6/ezH3Mhn46bJMTfeVTtcYMg=="
},
"@types/qs": {
"version": "6.9.8",
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.9.8.tgz",
@ -15007,6 +15047,14 @@
"requires": {
"normalize-path": "^3.0.0",
"picomatch": "^2.0.4"
},
"dependencies": {
"picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true
}
}
},
"app-root-path": {
@ -17938,6 +17986,14 @@
"ci-info": "^3.2.0",
"graceful-fs": "^4.2.9",
"picomatch": "^2.2.3"
},
"dependencies": {
"picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true
}
}
},
"jest-validate": {
@ -18339,6 +18395,14 @@
"requires": {
"braces": "^3.0.2",
"picomatch": "^2.3.1"
},
"dependencies": {
"picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true
}
}
},
"mime": {
@ -19005,10 +19069,9 @@
"dev": true
},
"picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-3.0.1.tgz",
"integrity": "sha512-I3EurrIQMlRc9IaAZnqRR044Phh2DXY+55o7uJ0V+hYZAcQYSuFWsc9q5PvyDHUSCe1Qxn/iBz+78s86zWnGag=="
},
"pirates": {
"version": "4.0.6",
@ -19345,6 +19408,14 @@
"dev": true,
"requires": {
"picomatch": "^2.2.1"
},
"dependencies": {
"picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true
}
}
},
"rechoir": {

3
server/package.json
View File

@ -47,6 +47,7 @@
"@nestjs/typeorm": "^10.0.0",
"@nestjs/websockets": "^10.2.2",
"@socket.io/postgres-adapter": "^0.3.1",
"@types/picomatch": "^2.3.3",
"archiver": "^6.0.0",
"async-lock": "^1.4.0",
"axios": "^1.5.0",
@ -70,6 +71,7 @@
"node-addon-api": "^7.0.0",
"openid-client": "^5.4.3",
"pg": "^8.11.3",
"picomatch": "^3.0.1",
"reflect-metadata": "^0.1.13",
"rxjs": "^7.8.1",
"sanitize-filename": "^1.6.3",
@ -101,6 +103,7 @@
"@types/ua-parser-js": "^0.7.36",
"@typescript-eslint/eslint-plugin": "^6.4.1",
"@typescript-eslint/parser": "^6.4.1",
"chokidar": "^3.5.3",
"dotenv": "^16.3.1",
"eslint": "^8.48.0",
"eslint-config-prettier": "^9.0.0",

4
server/src/domain/library/library.dto.ts
View File

@ -26,6 +26,10 @@ export class CreateLibraryDto {
@IsString({ each: true })
@IsNotEmpty({ each: true })
exclusionPatterns?: string[];
@IsOptional()
@IsBoolean()
isWatched?: boolean;
}
export class UpdateLibraryDto {

560
server/src/domain/library/library.service.spec.ts
View File

@ -2,9 +2,9 @@ import { AssetType, LibraryType, SystemConfig, SystemConfigKey, UserEntity } fro
import { BadRequestException } from '@nestjs/common';
import {
IAccessRepositoryMock,
assetStub,
authStub,
IAccessRepositoryMock,
libraryStub,
newAccessRepositoryMock,
newAssetRepositoryMock,
@ -14,8 +14,11 @@ import {
newStorageRepositoryMock,
newSystemConfigRepositoryMock,
newUserRepositoryMock,
systemConfigStub,
userStub,
} from '@test';
import { newFSWatcherMock } from '@test/mocks';
import { Stats } from 'fs';
import { ILibraryFileJob, ILibraryRefreshJob, JobName } from '../job';
import {
@ -28,6 +31,7 @@ import {
IUserRepository,
} from '../repositories';
import { SystemConfigCore } from '../system-config/system-config.core';
import { mapLibrary } from './library.dto';
import { LibraryService } from './library.service';
describe(LibraryService.name, () => {
@ -94,11 +98,60 @@ describe(LibraryService.name, () => {
enabled: true,
cronExpression: '0 1 * * *',
},
watch: { enabled: false },
},
} as SystemConfig);
expect(jobMock.updateCronJob).toHaveBeenCalledWith('libraryScan', '0 1 * * *', true);
});
it('should initialize watcher for all external libraries', async () => {
libraryMock.getAll.mockResolvedValue([
libraryStub.externalLibraryWithImportPaths1,
libraryStub.externalLibraryWithImportPaths2,
]);
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
libraryMock.get.mockImplementation(async (id) => {
switch (id) {
case libraryStub.externalLibraryWithImportPaths1.id:
return libraryStub.externalLibraryWithImportPaths1;
case libraryStub.externalLibraryWithImportPaths2.id:
return libraryStub.externalLibraryWithImportPaths2;
default:
return null;
}
});
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await sut.init();
expect(storageMock.watch.mock.calls).toEqual(
expect.arrayContaining([
(libraryStub.externalLibrary1.importPaths, expect.anything()),
(libraryStub.externalLibrary2.importPaths, expect.anything()),
]),
);
});
it('should not initialize when watching is disabled', async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchDisabled);
await sut.init();
expect(storageMock.watch).not.toHaveBeenCalled();
});
});
describe('handleQueueAssetRefresh', () => {
@ -148,6 +201,34 @@ describe(LibraryService.name, () => {
]);
});
it('should force queue new assets', async () => {
const mockLibraryJob: ILibraryRefreshJob = {
id: libraryStub.externalLibrary1.id,
refreshModifiedFiles: false,
refreshAllFiles: true,
};
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
storageMock.crawl.mockResolvedValue(['/data/user1/photo.jpg']);
assetMock.getByLibraryId.mockResolvedValue([]);
libraryMock.getOnlineAssetPaths.mockResolvedValue([]);
userMock.get.mockResolvedValue(userStub.externalPath1);
await sut.handleQueueAssetRefresh(mockLibraryJob);
expect(jobMock.queueAll).toHaveBeenCalledWith([
{
name: JobName.LIBRARY_SCAN_ASSET,
data: {
id: libraryStub.externalLibrary1.id,
ownerId: libraryStub.externalLibrary1.owner.id,
assetPath: '/data/user1/photo.jpg',
force: true,
},
},
]);
});
it("should mark assets outside of the user's external path as offline", async () => {
const mockLibraryJob: ILibraryRefreshJob = {
id: libraryStub.externalLibrary1.id,
@ -564,7 +645,7 @@ describe(LibraryService.name, () => {
expect(createdAsset.fileModifiedAt).toEqual(filemtime);
});
it('should error when asset does not exist', async () => {
it('should throw error when asset does not exist', async () => {
storageMock.stat.mockRejectedValue(new Error("ENOENT, no such file or directory '/data/user1/photo.jpg'"));
const mockLibraryJob: ILibraryFileJob = {
@ -625,6 +706,31 @@ describe(LibraryService.name, () => {
expect(libraryMock.softDelete).toHaveBeenCalledWith(libraryStub.externalLibrary1.id);
});
it('should unwatch an external library when deleted', async () => {
assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.image);
libraryMock.getUploadLibraryCount.mockResolvedValue(1);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await sut.init();
await sut.delete(authStub.admin, libraryStub.externalLibraryWithImportPaths1.id);
expect(mockWatcher.close).toHaveBeenCalled();
});
});
describe('getCount', () => {
@ -638,7 +744,7 @@ describe(LibraryService.name, () => {
});
describe('get', () => {
it('can return a library', async () => {
it('should return a library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.uploadLibrary1);
await expect(sut.get(authStub.admin, libraryStub.uploadLibrary1.id)).resolves.toEqual(
expect.objectContaining({
@ -659,7 +765,7 @@ describe(LibraryService.name, () => {
});
describe('getAllForUser', () => {
it('can return all libraries for user', async () => {
it('should return all libraries for user', async () => {
libraryMock.getAllByUserId.mockResolvedValue([libraryStub.uploadLibrary1, libraryStub.externalLibrary1]);
await expect(sut.getAllForUser(authStub.admin)).resolves.toEqual([
expect.objectContaining({
@ -679,7 +785,7 @@ describe(LibraryService.name, () => {
});
describe('getStatistics', () => {
it('can return library statistics', async () => {
it('should return library statistics', async () => {
libraryMock.getStatistics.mockResolvedValue({ photos: 10, videos: 0, total: 10, usage: 1337 });
await expect(sut.getStatistics(authStub.admin, libraryStub.uploadLibrary1.id)).resolves.toEqual({
photos: 10,
@ -694,7 +800,7 @@ describe(LibraryService.name, () => {
describe('create', () => {
describe('external library', () => {
it('can create with default settings', async () => {
it('should create with default settings', async () => {
libraryMock.create.mockResolvedValue(libraryStub.externalLibrary1);
await expect(
sut.create(authStub.admin, {
@ -717,7 +823,7 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).toHaveBeenCalledWith(
expect.objectContaining({
name: 'New External Library',
name: expect.any(String),
type: LibraryType.EXTERNAL,
importPaths: [],
exclusionPatterns: [],
@ -726,7 +832,7 @@ describe(LibraryService.name, () => {
);
});
it('can create with name', async () => {
it('should create with name', async () => {
libraryMock.create.mockResolvedValue(libraryStub.externalLibrary1);
await expect(
sut.create(authStub.admin, {
@ -759,7 +865,7 @@ describe(LibraryService.name, () => {
);
});
it('can create invisible', async () => {
it('should create invisible', async () => {
libraryMock.create.mockResolvedValue(libraryStub.externalLibrary1);
await expect(
sut.create(authStub.admin, {
@ -783,7 +889,7 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).toHaveBeenCalledWith(
expect.objectContaining({
name: 'New External Library',
name: expect.any(String),
type: LibraryType.EXTERNAL,
importPaths: [],
exclusionPatterns: [],
@ -792,7 +898,7 @@ describe(LibraryService.name, () => {
);
});
it('can create with import paths', async () => {
it('should create with import paths', async () => {
libraryMock.create.mockResolvedValue(libraryStub.externalLibrary1);
await expect(
sut.create(authStub.admin, {
@ -816,7 +922,7 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).toHaveBeenCalledWith(
expect.objectContaining({
name: 'New External Library',
name: expect.any(String),
type: LibraryType.EXTERNAL,
importPaths: ['/data/images', '/data/videos'],
exclusionPatterns: [],
@ -825,7 +931,35 @@ describe(LibraryService.name, () => {
);
});
it('can create with exclusion patterns', async () => {
it('should create watched with import paths', async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.create.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([]);
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await sut.init();
await sut.create(authStub.admin, {
type: LibraryType.EXTERNAL,
importPaths: libraryStub.externalLibraryWithImportPaths1.importPaths,
});
expect(storageMock.watch).toHaveBeenCalledWith(
libraryStub.externalLibraryWithImportPaths1.importPaths,
expect.anything(),
);
});
it('should create with exclusion patterns', async () => {
libraryMock.create.mockResolvedValue(libraryStub.externalLibrary1);
await expect(
sut.create(authStub.admin, {
@ -849,7 +983,7 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).toHaveBeenCalledWith(
expect.objectContaining({
name: 'New External Library',
name: expect.any(String),
type: LibraryType.EXTERNAL,
importPaths: [],
exclusionPatterns: ['*.tmp', '*.bak'],
@ -860,7 +994,7 @@ describe(LibraryService.name, () => {
});
describe('upload library', () => {
it('can create with default settings', async () => {
it('should create with default settings', async () => {
libraryMock.create.mockResolvedValue(libraryStub.uploadLibrary1);
await expect(
sut.create(authStub.admin, {
@ -892,7 +1026,7 @@ describe(LibraryService.name, () => {
);
});
it('can create with name', async () => {
it('should create with name', async () => {
libraryMock.create.mockResolvedValue(libraryStub.uploadLibrary1);
await expect(
sut.create(authStub.admin, {
@ -925,7 +1059,7 @@ describe(LibraryService.name, () => {
);
});
it('can not create with import paths', async () => {
it('should not create with import paths', async () => {
await expect(
sut.create(authStub.admin, {
type: LibraryType.UPLOAD,
@ -936,7 +1070,7 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).not.toHaveBeenCalled();
});
it('can not create with exclusion patterns', async () => {
it('should not create with exclusion patterns', async () => {
await expect(
sut.create(authStub.admin, {
type: LibraryType.UPLOAD,
@ -946,11 +1080,22 @@ describe(LibraryService.name, () => {
expect(libraryMock.create).not.toHaveBeenCalled();
});
it('should not create watched', async () => {
await expect(
sut.create(authStub.admin, {
type: LibraryType.UPLOAD,
isWatched: true,
}),
).rejects.toBeInstanceOf(BadRequestException);
expect(storageMock.watch).not.toHaveBeenCalled();
});
});
});
describe('handleQueueCleanup', () => {
it('can queue cleanup jobs', async () => {
it('should queue cleanup jobs', async () => {
libraryMock.getAllDeleted.mockResolvedValue([libraryStub.uploadLibrary1, libraryStub.externalLibrary1]);
await expect(sut.handleQueueCleanup()).resolves.toBe(true);
@ -962,19 +1107,357 @@ describe(LibraryService.name, () => {
});
describe('update', () => {
it('can update library ', async () => {
beforeEach(async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.getAll.mockResolvedValue([]);
await sut.init();
});
it('should update library', async () => {
libraryMock.update.mockResolvedValue(libraryStub.uploadLibrary1);
await expect(sut.update(authStub.admin, authStub.admin.user.id, {})).resolves.toBeTruthy();
await expect(sut.update(authStub.admin, authStub.admin.user.id, {})).resolves.toEqual(
mapLibrary(libraryStub.uploadLibrary1),
);
expect(libraryMock.update).toHaveBeenCalledWith(
expect.objectContaining({
id: authStub.admin.user.id,
}),
);
});
it('should re-watch library when updating import paths', async () => {
libraryMock.update.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await expect(sut.update(authStub.admin, authStub.admin.user.id, { importPaths: ['/foo'] })).resolves.toEqual(
mapLibrary(libraryStub.externalLibraryWithImportPaths1),
);
expect(libraryMock.update).toHaveBeenCalledWith(
expect.objectContaining({
id: authStub.admin.user.id,
}),
);
expect(storageMock.watch).toHaveBeenCalledWith(
libraryStub.externalLibraryWithImportPaths1.importPaths,
expect.anything(),
);
});
it('should re-watch library when updating exclusion patterns', async () => {
libraryMock.update.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await expect(sut.update(authStub.admin, authStub.admin.user.id, { exclusionPatterns: ['bar'] })).resolves.toEqual(
mapLibrary(libraryStub.externalLibraryWithImportPaths1),
);
expect(libraryMock.update).toHaveBeenCalledWith(
expect.objectContaining({
id: authStub.admin.user.id,
}),
);
expect(storageMock.watch).toHaveBeenCalledWith(expect.arrayContaining([expect.any(String)]), expect.anything());
});
});
describe('watchAll new', () => {
describe('watching disabled', () => {
beforeEach(async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchDisabled);
await sut.init();
});
it('should not watch library', async () => {
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
await sut.watchAll();
expect(storageMock.watch).not.toHaveBeenCalled();
});
});
describe('watching enabled', () => {
const mockWatcher = newFSWatcherMock();
beforeEach(async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.getAll.mockResolvedValue([]);
await sut.init();
storageMock.watch.mockReturnValue(mockWatcher);
});
it('should watch library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
const mockWatcher = newFSWatcherMock();
let isReady = false;
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
isReady = true;
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await sut.watchAll();
expect(storageMock.watch).toHaveBeenCalledWith(
libraryStub.externalLibraryWithImportPaths1.importPaths,
expect.anything(),
);
expect(isReady).toBe(true);
});
it('should watch and unwatch library', async () => {
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
await sut.watchAll();
await sut.unwatch(libraryStub.externalLibraryWithImportPaths1.id);
expect(mockWatcher.close).toHaveBeenCalled();
});
it('should not watch library without import paths', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibrary1]);
await sut.watchAll();
expect(storageMock.watch).not.toHaveBeenCalled();
});
it('should throw error when watching upload library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.uploadLibrary1);
libraryMock.getAll.mockResolvedValue([libraryStub.uploadLibrary1]);
await expect(sut.watchAll()).rejects.toThrow('Can only watch external libraries');
expect(storageMock.watch).not.toHaveBeenCalled();
});
it('should handle a new file event', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'add') {
callback('/foo/photo.jpg');
}
});
await sut.watchAll();
expect(jobMock.queueAll).toHaveBeenCalledWith([
{
name: JobName.LIBRARY_SCAN_ASSET,
data: {
id: libraryStub.externalLibraryWithImportPaths1.id,
assetPath: '/foo/photo.jpg',
ownerId: libraryStub.externalLibraryWithImportPaths1.owner.id,
force: false,
},
},
]);
});
it('should handle a file change event', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'change') {
callback('/foo/photo.jpg');
}
});
await sut.watchAll();
expect(jobMock.queueAll).toHaveBeenCalledWith([
{
name: JobName.LIBRARY_SCAN_ASSET,
data: {
id: libraryStub.externalLibraryWithImportPaths1.id,
assetPath: '/foo/photo.jpg',
ownerId: libraryStub.externalLibraryWithImportPaths1.owner.id,
force: false,
},
},
]);
});
it('should handle a file unlink event', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.external);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'unlink') {
callback('/foo/photo.jpg');
}
});
await sut.watchAll();
expect(assetMock.save).toHaveBeenCalledWith({ id: assetStub.external.id, isOffline: true });
});
it('should handle an error event', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.external);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
let didError = false;
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'error') {
didError = true;
callback('Error!');
}
});
await sut.watchAll();
expect(didError).toBe(true);
});
it('should ignore unknown extensions', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'add') {
callback('/foo/photo.txt');
}
});
await sut.watchAll();
expect(jobMock.queue).not.toHaveBeenCalled();
});
it('should ignore excluded paths', async () => {
libraryMock.get.mockResolvedValue(libraryStub.patternPath);
libraryMock.getAll.mockResolvedValue([libraryStub.patternPath]);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'add') {
callback('/dir1/photo.txt');
}
});
await sut.watchAll();
expect(jobMock.queue).not.toHaveBeenCalled();
});
it('should ignore excluded paths without case sensitivity', async () => {
libraryMock.get.mockResolvedValue(libraryStub.patternPath);
libraryMock.getAll.mockResolvedValue([libraryStub.patternPath]);
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
} else if (event === 'add') {
callback('/DIR1/photo.txt');
}
});
await sut.watchAll();
expect(jobMock.queue).not.toHaveBeenCalled();
});
});
});
describe('tearDown', () => {
it('should tear down all watchers', async () => {
libraryMock.getAll.mockResolvedValue([
libraryStub.externalLibraryWithImportPaths1,
libraryStub.externalLibraryWithImportPaths2,
]);
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchEnabled);
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
libraryMock.get.mockImplementation(async (id) => {
switch (id) {
case libraryStub.externalLibraryWithImportPaths1.id:
return libraryStub.externalLibraryWithImportPaths1;
case libraryStub.externalLibraryWithImportPaths2.id:
return libraryStub.externalLibraryWithImportPaths2;
default:
return null;
}
});
const mockWatcher = newFSWatcherMock();
mockWatcher.on.mockImplementation((event, callback) => {
if (event === 'ready') {
callback();
}
});
storageMock.watch.mockReturnValue(mockWatcher);
await sut.init();
await sut.unwatchAll();
expect(mockWatcher.close).toHaveBeenCalledTimes(2);
});
});
describe('handleDeleteLibrary', () => {
it('can not delete a nonexistent library', async () => {
it('should not delete a nonexistent library', async () => {
libraryMock.get.mockImplementation(async () => {
return null;
});
@ -984,7 +1467,7 @@ describe(LibraryService.name, () => {
await expect(sut.handleDeleteLibrary({ id: libraryStub.uploadLibrary1.id })).resolves.toBe(false);
});
it('can delete an empty library', async () => {
it('should delete an empty library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.uploadLibrary1);
libraryMock.getAssetIds.mockResolvedValue([]);
libraryMock.delete.mockImplementation(async () => {});
@ -992,7 +1475,7 @@ describe(LibraryService.name, () => {
await expect(sut.handleDeleteLibrary({ id: libraryStub.uploadLibrary1.id })).resolves.toBe(true);
});
it('can delete a library with assets', async () => {
it('should delete a library with assets', async () => {
libraryMock.get.mockResolvedValue(libraryStub.uploadLibrary1);
libraryMock.getAssetIds.mockResolvedValue([assetStub.image1.id]);
libraryMock.delete.mockImplementation(async () => {});
@ -1004,7 +1487,7 @@ describe(LibraryService.name, () => {
});
describe('queueScan', () => {
it('can queue a library scan of external library', async () => {
it('should queue a library scan of external library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
await sut.queueScan(authStub.admin, libraryStub.externalLibrary1.id, {});
@ -1023,7 +1506,7 @@ describe(LibraryService.name, () => {
]);
});
it('can not queue a library scan of upload library', async () => {
it('should not queue a library scan of upload library', async () => {
libraryMock.get.mockResolvedValue(libraryStub.uploadLibrary1);
await expect(sut.queueScan(authStub.admin, libraryStub.uploadLibrary1.id, {})).rejects.toBeInstanceOf(
@ -1033,7 +1516,7 @@ describe(LibraryService.name, () => {
expect(jobMock.queue).not.toBeCalled();
});
it('can queue a library scan of all modified assets', async () => {
it('should queue a library scan of all modified assets', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
await sut.queueScan(authStub.admin, libraryStub.externalLibrary1.id, { refreshModifiedFiles: true });
@ -1052,7 +1535,7 @@ describe(LibraryService.name, () => {
]);
});
it('can queue a forced library scan', async () => {
it('should queue a forced library scan', async () => {
libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1);
await sut.queueScan(authStub.admin, libraryStub.externalLibrary1.id, { refreshAllFiles: true });
@ -1073,7 +1556,7 @@ describe(LibraryService.name, () => {
});
describe('queueEmptyTrash', () => {
it('can queue the trash job', async () => {
it('should queue the trash job', async () => {
await sut.queueRemoveOffline(authStub.admin, libraryStub.externalLibrary1.id);
expect(jobMock.queue.mock.calls).toEqual([
@ -1090,7 +1573,7 @@ describe(LibraryService.name, () => {
});
describe('handleQueueAllScan', () => {
it('can queue the refresh job', async () => {
it('should queue the refresh job', async () => {
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibrary1]);
await expect(sut.handleQueueAllScan({})).resolves.toBe(true);
@ -1115,19 +1598,16 @@ describe(LibraryService.name, () => {
]);
});
it('can queue the force refresh job', async () => {
it('should queue the force refresh job', async () => {
libraryMock.getAll.mockResolvedValue([libraryStub.externalLibrary1]);
await expect(sut.handleQueueAllScan({ force: true })).resolves.toBe(true);
expect(jobMock.queue.mock.calls).toEqual([
[
{
name: JobName.LIBRARY_QUEUE_CLEANUP,
data: {},
},
],
]);
expect(jobMock.queue).toHaveBeenCalledWith({
name: JobName.LIBRARY_QUEUE_CLEANUP,
data: {},
});
expect(jobMock.queueAll).toHaveBeenCalledWith([
{
name: JobName.LIBRARY_SCAN,
@ -1142,7 +1622,7 @@ describe(LibraryService.name, () => {
});
describe('handleRemoveOfflineFiles', () => {
it('can queue trash deletion jobs', async () => {
it('should queue trash deletion jobs', async () => {
assetMock.getWith.mockResolvedValue({ items: [assetStub.image1], hasNextPage: false });
assetMock.getById.mockResolvedValue(assetStub.image1);

181
server/src/domain/library/library.service.ts
View File

@ -1,5 +1,7 @@
import { AssetType, LibraryType } from '@app/infra/entities';
import { BadRequestException, Inject, Injectable } from '@nestjs/common';
import picomatch from 'picomatch';
import { R_OK } from 'node:constants';
import { Stats } from 'node:fs';
import path from 'node:path';
@ -11,6 +13,7 @@ import { usePagination, validateCronExpression } from '../domain.util';
import { IBaseJob, IEntityJob, ILibraryFileJob, ILibraryRefreshJob, JOBS_ASSET_PAGINATION_SIZE, JobName } from '../job';
import { ImmichLogger } from '@app/infra/logger';
import { EventEmitter } from 'events';
import {
IAccessRepository,
IAssetRepository,
@ -33,11 +36,15 @@ import {
} from './library.dto';
@Injectable()
export class LibraryService {
export class LibraryService extends EventEmitter {
readonly logger = new ImmichLogger(LibraryService.name);
private access: AccessCore;
private configCore: SystemConfigCore;
private watchLibraries = false;
private watchers: Record<string, () => Promise<void>> = {};
constructor(
@Inject(IAccessRepository) accessRepository: IAccessRepository,
@Inject(IAssetRepository) private assetRepository: IAssetRepository,
@ -48,6 +55,7 @@ export class LibraryService {
@Inject(IStorageRepository) private storageRepository: IStorageRepository,
@Inject(IUserRepository) private userRepository: IUserRepository,
) {
super();
this.access = AccessCore.create(accessRepository);
this.configCore = SystemConfigCore.create(configRepository);
this.configCore.addValidator((config) => {
@ -59,6 +67,7 @@ export class LibraryService {
async init() {
const config = await this.configCore.getConfig();
this.watchLibraries = config.library.watch.enabled;
this.jobRepository.addCronJob(
'libraryScan',
config.library.scan.cronExpression,
@ -66,11 +75,128 @@ export class LibraryService {
config.library.scan.enabled,
);
this.configCore.config$.subscribe((config) => {
if (this.watchLibraries) {
await this.watchAll();
}
this.configCore.config$.subscribe(async (config) => {
this.jobRepository.updateCronJob('libraryScan', config.library.scan.cronExpression, config.library.scan.enabled);
if (config.library.watch.enabled !== this.watchLibraries) {
this.watchLibraries = config.library.watch.enabled;
if (this.watchLibraries) {
await this.watchAll();
} else {
await this.unwatchAll();
}
}
});
}
private async watch(id: string): Promise<boolean> {
if (!this.watchLibraries) {
return false;
}
const library = await this.findOrFail(id);
if (library.type !== LibraryType.EXTERNAL) {
throw new BadRequestException('Can only watch external libraries');
} else if (library.importPaths.length === 0) {
return false;
}
await this.unwatch(id);
this.logger.log(`Starting to watch library ${library.id} with import path(s) ${library.importPaths}`);
const matcher = picomatch(`**/*{${mimeTypes.getSupportedFileExtensions().join(',')}}`, {
nocase: true,
ignore: library.exclusionPatterns,
});
const config = await this.configCore.getConfig();
this.logger.debug(
`Settings for watcher: usePolling: ${config.library.watch.usePolling}, interval: ${config.library.watch.interval}`,
);
const watcher = this.storageRepository.watch(library.importPaths, {
usePolling: config.library.watch.usePolling,
interval: config.library.watch.interval,
binaryInterval: config.library.watch.interval,
ignoreInitial: true,
});
this.watchers[id] = async () => {
await watcher.close();
};
watcher.on('add', async (path) => {
this.logger.debug(`File add event received for ${path} in library ${library.id}}`);
if (matcher(path)) {
await this.scanAssets(library.id, [path], library.ownerId, false);
}
this.emit('add', path);
});
watcher.on('change', async (path) => {
this.logger.debug(`Detected file change for ${path} in library ${library.id}`);
if (matcher(path)) {
// Note: if the changed file was not previously imported, it will be imported now.
await this.scanAssets(library.id, [path], library.ownerId, false);
}
this.emit('change', path);
});
watcher.on('unlink', async (path) => {
this.logger.debug(`Detected deleted file at ${path} in library ${library.id}`);
const existingAssetEntity = await this.assetRepository.getByLibraryIdAndOriginalPath(library.id, path);
if (existingAssetEntity && matcher(path)) {
await this.assetRepository.save({ id: existingAssetEntity.id, isOffline: true });
}
this.emit('unlink', path);
});
watcher.on('error', async (error) => {
// TODO: should we log, or throw an exception?
this.logger.error(`Library watcher for library ${library.id} encountered error: ${error}`);
});
// Wait for the watcher to initialize before returning
await new Promise<void>((resolve) => {
watcher.on('ready', async () => {
resolve();
});
});
return true;
}
async unwatch(id: string) {
if (this.watchers.hasOwnProperty(id)) {
await this.watchers[id]();
delete this.watchers[id];
}
}
async unwatchAll() {
for (const id in this.watchers) {
await this.unwatch(id);
}
}
async watchAll() {
const libraries = await this.repository.getAll(false, LibraryType.EXTERNAL);
for (const library of libraries) {
await this.watch(library.id);
}
}
async getStatistics(auth: AuthDto, id: string): Promise<LibraryStatsResponseDto> {
await this.access.requirePermission(auth, Permission.LIBRARY_READ, id);
return this.repository.getStatistics(id);
@ -117,6 +243,9 @@ export class LibraryService {
if (dto.exclusionPatterns && dto.exclusionPatterns.length > 0) {
throw new BadRequestException('Upload libraries cannot have exclusion patterns');
}
if (dto.isWatched) {
throw new BadRequestException('Upload libraries cannot be watched');
}
break;
}
@ -129,12 +258,38 @@ export class LibraryService {
isVisible: dto.isVisible ?? true,
});
this.logger.log(`Creating ${dto.type} library for user ${auth.user.name}`);
if (dto.type === LibraryType.EXTERNAL && this.watchLibraries) {
await this.watch(library.id);
}
return mapLibrary(library);
}
private async scanAssets(libraryId: string, assetPaths: string[], ownerId: string, force = false) {
await this.jobRepository.queueAll(
assetPaths.map((assetPath) => ({
name: JobName.LIBRARY_SCAN_ASSET,
data: {
id: libraryId,
assetPath: path.normalize(assetPath),
ownerId,
force,
},
})),
);
}
async update(auth: AuthDto, id: string, dto: UpdateLibraryDto): Promise<LibraryResponseDto> {
await this.access.requirePermission(auth, Permission.LIBRARY_UPDATE, id);
const library = await this.repository.update({ id, ...dto });
if (dto.importPaths || dto.exclusionPatterns) {
// Re-watch library to use new paths and/or exclusion patterns
await this.watch(id);
}
return mapLibrary(library);
}
@ -147,6 +302,10 @@ export class LibraryService {
throw new BadRequestException('Cannot delete the last upload library');
}
if (this.watchLibraries) {
await this.unwatch(id);
}
await this.repository.softDelete(id);
await this.jobRepository.queue({ name: JobName.LIBRARY_DELETE, data: { id } });
}
@ -245,8 +404,6 @@ export class LibraryService {
const deviceAssetId = `${basename(assetPath)}`.replace(/\s+/g, '');
const pathHash = this.cryptoRepository.hashSha1(`path:${assetPath}`);
let assetId;
if (doImport) {
const library = await this.repository.get(job.id, true);
@ -255,6 +412,8 @@ export class LibraryService {
return false;
}
const pathHash = this.cryptoRepository.hashSha1(`path:${assetPath}`);
// TODO: In wait of refactoring the domain asset service, this function is just manually written like this
const addedAsset = await this.assetRepository.create({
ownerId: job.ownerId,
@ -387,7 +546,7 @@ export class LibraryService {
assetPath.match(new RegExp(`^${user.externalPath}`)),
);
this.logger.debug(`Found ${crawledAssetPaths.length} assets when crawling import paths ${library.importPaths}`);
this.logger.debug(`Found ${crawledAssetPaths.length} asset(s) when crawling import paths ${library.importPaths}`);
const assetsInLibrary = await this.assetRepository.getByLibraryId([job.id]);
const onlineFiles = new Set(crawledAssetPaths);
const offlineAssetIds = assetsInLibrary
@ -411,17 +570,7 @@ export class LibraryService {
this.logger.debug(`Will import ${filteredPaths.length} new asset(s)`);
}
await this.jobRepository.queueAll(
filteredPaths.map((assetPath) => ({
name: JobName.LIBRARY_SCAN_ASSET,
data: {
id: job.id,
assetPath: path.normalize(assetPath),
ownerId: library.ownerId,
force: job.refreshAllFiles ?? false,
},
})),
);
await this.scanAssets(job.id, filteredPaths, library.ownerId, job.refreshAllFiles ?? false);
}
await this.repository.update({ id: job.id, refreshedAt: new Date() });

2
server/src/domain/repositories/job.repository.ts
View File

@ -78,7 +78,7 @@ export type JobItem =
// Filesystem
| { name: JobName.DELETE_FILES; data: IDeleteFilesJob }
// Audit log cleanup
// Audit Log Cleanup
| { name: JobName.CLEAN_OLD_AUDIT_LOGS; data?: IBaseJob }
// Asset Deletion

4
server/src/domain/repositories/storage.repository.ts
View File

@ -1,3 +1,4 @@
import { FSWatcher, WatchOptions } from 'chokidar';
import { Stats } from 'fs';
import { FileReadOptions } from 'fs/promises';
import { Readable } from 'stream';
@ -22,6 +23,8 @@ export interface DiskUsage {
export const IStorageRepository = 'IStorageRepository';
export interface ImmichWatcher extends FSWatcher {}
export interface IStorageRepository {
createZipStream(): ImmichZipStream;
createReadStream(filepath: string, mimeType?: string | null): Promise<ImmichReadStream>;
@ -38,4 +41,5 @@ export interface IStorageRepository {
crawl(crawlOptions: CrawlOptionsDto): Promise<string[]>;
copyFile(source: string, target: string): Promise<void>;
rename(source: string, target: string): Promise<void>;
watch(paths: string[], options: WatchOptions): ImmichWatcher;
}

21
server/src/domain/system-config/dto/system-config-library.dto.ts
View File

@ -1,9 +1,12 @@
import { validateCronExpression } from '@app/domain';
import { ApiProperty } from '@nestjs/swagger';
import { Type } from 'class-transformer';
import {
IsBoolean,
IsInt,
IsNotEmpty,
IsObject,
IsPositive,
IsString,
Validate,
ValidateIf,
@ -32,9 +35,27 @@ export class SystemConfigLibraryScanDto {
cronExpression!: string;
}
export class SystemConfigLibraryWatchDto {
@IsBoolean()
enabled!: boolean;
@IsBoolean()
usePolling!: boolean;
@IsInt()
@IsPositive()
@ApiProperty({ type: 'integer' })
interval!: number;
}
export class SystemConfigLibraryDto {
@Type(() => SystemConfigLibraryScanDto)
@ValidateNested()
@IsObject()
scan!: SystemConfigLibraryScanDto;
@Type(() => SystemConfigLibraryWatchDto)
@ValidateNested()
@IsObject()
watch!: SystemConfigLibraryWatchDto;
}

5
server/src/domain/system-config/system-config.core.ts
View File

@ -129,6 +129,11 @@ export const defaults = Object.freeze<SystemConfig>({
enabled: true,
cronExpression: CronExpression.EVERY_DAY_AT_MIDNIGHT,
},
watch: {
enabled: false,
usePolling: false,
interval: 10000,
},
},
server: {
externalDomain: '',

5
server/src/domain/system-config/system-config.service.spec.ts
View File

@ -133,6 +133,11 @@ const updatedConfig = Object.freeze<SystemConfig>({
enabled: true,
cronExpression: '0 0 * * *',
},
watch: {
enabled: false,
usePolling: false,
interval: 10000,
},
},
});

4
server/src/immich/app.service.ts
View File

@ -71,6 +71,10 @@ export class AppService {
this.logger.log(`Feature Flags: ${JSON.stringify(await this.serverService.getFeatures(), null, 2)}`);
}
async teardown() {
await this.libraryService.unwatchAll();
}
ssr(excludePaths: string[]) {
let index = '';
try {

12
server/src/infra/entities/system-config.entity.ts
View File

@ -50,6 +50,13 @@ export enum SystemConfigKey {
LIBRARY_SCAN_ENABLED = 'library.scan.enabled',
LIBRARY_SCAN_CRON_EXPRESSION = 'library.scan.cronExpression',
LIBRARY_WATCH_ENABLED = 'library.watch.enabled',
LIBRARY_WATCH_USE_POLLING = 'library.watch.usePolling',
LIBRARY_WATCH_INTERVAL = 'library.watch.interval',
LIBRARY_WATCH_BINARY_INTERVAL = 'library.watch.binaryInterval',
LIBRARY_WATCH_WRITE_STABILITY_THRESHOLD = 'library.watch.awaitWriteFinish.stabilityThreshold',
LIBRARY_WATCH_WRITE_POLL_INTERVAL = 'library.watch.awaitWriteFinish.pollInterval',
LOGGING_ENABLED = 'logging.enabled',
LOGGING_LEVEL = 'logging.level',
@ -253,6 +260,11 @@ export interface SystemConfig {
enabled: boolean;
cronExpression: string;
};
watch: {
enabled: boolean;
usePolling: boolean;
interval: number;
};
};
server: {
externalDomain: string;

6
server/src/infra/repositories/filesystem.provider.spec.ts
View File

@ -180,7 +180,11 @@ const tests: Test[] = [
];
describe(FilesystemProvider.name, () => {
const sut = new FilesystemProvider();
let sut: FilesystemProvider;
beforeEach(() => {
sut = new FilesystemProvider();
});
afterEach(() => {
mockfs.restore();

6
server/src/infra/repositories/filesystem.provider.ts
View File

@ -2,12 +2,14 @@ import {
CrawlOptionsDto,
DiskUsage,
ImmichReadStream,
ImmichWatcher,
ImmichZipStream,
IStorageRepository,
mimeTypes,
} from '@app/domain';
import { ImmichLogger } from '@app/infra/logger';
import archiver from 'archiver';
import chokidar, { WatchOptions } from 'chokidar';
import { constants, createReadStream, existsSync, mkdirSync } from 'fs';
import fs, { copyFile, readdir, rename, writeFile } from 'fs/promises';
import { glob } from 'glob';
@ -132,5 +134,9 @@ export class FilesystemProvider implements IStorageRepository {
});
}
watch(paths: string[], options: WatchOptions): ImmichWatcher {
return chokidar.watch(paths, options);
}
readdir = readdir;
}

4
server/src/microservices/app.service.ts
View File

@ -3,6 +3,7 @@ import {
AuditService,
DatabaseService,
IDeleteFilesJob,
IStorageRepository,
JobName,
JobService,
LibraryService,
@ -15,7 +16,7 @@ import {
SystemConfigService,
UserService,
} from '@app/domain';
import { Injectable } from '@nestjs/common';
import { Inject, Injectable } from '@nestjs/common';
@Injectable()
export class AppService {
@ -33,6 +34,7 @@ export class AppService {
private storageService: StorageService,
private userService: UserService,
private databaseService: DatabaseService,
@Inject(IStorageRepository) private storageRepository: IStorageRepository,
) {}
async init() {

32
server/src/test-utils/utils.ts
View File

@ -8,8 +8,10 @@ import { DateTime } from 'luxon';
import * as fs from 'node:fs';
import path from 'node:path';
import { Server } from 'node:tls';
import { EventEmitter } from 'stream';
import { EntityTarget, ObjectLiteral } from 'typeorm';
import { AppService } from '../microservices/app.service';
import { AppService } from '../immich/app.service';
import { AppService as MicroAppService } from '../microservices/app.service';
export const IMMICH_TEST_ASSET_PATH = process.env.IMMICH_TEST_ASSET_PATH as string;
export const IMMICH_TEST_ASSET_TEMP_PATH = path.normalize(`${IMMICH_TEST_ASSET_PATH}/temp/`);
@ -95,7 +97,10 @@ let app: INestApplication;
export const testApp = {
create: async (): Promise<INestApplication> => {
const moduleFixture = await Test.createTestingModule({ imports: [AppModule], providers: [AppService] })
const moduleFixture = await Test.createTestingModule({
imports: [AppModule],
providers: [AppService, MicroAppService],
})
.overrideModule(InfraModule)
.useModule(InfraTestModule)
.overrideProvider(IJobRepository)
@ -106,7 +111,9 @@ export const testApp = {
app = await moduleFixture.createNestApplication().init();
await app.listen(0);
await db.reset();
await app.get(AppService).init();
await app.get(MicroAppService).init();
const port = app.getHttpServer().address().port;
const protocol = app instanceof Server ? 'https' : 'http';
@ -115,11 +122,15 @@ export const testApp = {
return app;
},
reset: async (options?: ResetOptions) => {
await app.get(AppService).init();
await db.reset(options);
await app.get(AppService).init();
await app.get(MicroAppService).init();
},
get: (member: any) => app.get(member),
teardown: async () => {
if (app) {
await app.get(MicroAppService).teardown();
await app.get(AppService).teardown();
await app.close();
}
@ -127,6 +138,21 @@ export const testApp = {
},
};
export function waitForEvent<T>(emitter: EventEmitter, event: string): Promise<T> {
return new Promise((resolve, reject) => {
const success = (val: T) => {
emitter.off('error', fail);
resolve(val);
};
const fail = (err: Error) => {
emitter.off(event, success);
reject(err);
};
emitter.once(event, success);
emitter.once('error', fail);
});
}
const directoryExists = async (dirPath: string) =>
await fs.promises
.access(dirPath)

70
server/test/fixtures/library.stub.ts vendored
View File

@ -30,4 +30,74 @@ export const libraryStub = {
isVisible: true,
exclusionPatterns: [],
}),
externalLibrary2: Object.freeze<LibraryEntity>({
id: 'library-id2',
name: 'test_library2',
assets: [],
owner: userStub.externalPath1,
ownerId: 'user-id',
type: LibraryType.EXTERNAL,
importPaths: [],
createdAt: new Date('2021-01-01'),
updatedAt: new Date('2022-01-01'),
refreshedAt: null,
isVisible: true,
exclusionPatterns: [],
}),
externalLibraryWithImportPaths1: Object.freeze<LibraryEntity>({
id: 'library-id-with-paths1',
name: 'library-with-import-paths1',
assets: [],
owner: userStub.externalPath1,
ownerId: 'user-id',
type: LibraryType.EXTERNAL,
importPaths: ['/foo', '/bar'],
createdAt: new Date('2023-01-01'),
updatedAt: new Date('2023-01-01'),
refreshedAt: null,
isVisible: true,
exclusionPatterns: [],
}),
externalLibraryWithImportPaths2: Object.freeze<LibraryEntity>({
id: 'library-id-with-paths2',
name: 'library-with-import-paths2',
assets: [],
owner: userStub.externalPath1,
ownerId: 'user-id',
type: LibraryType.EXTERNAL,
importPaths: ['/xyz', '/asdf'],
createdAt: new Date('2023-01-01'),
updatedAt: new Date('2023-01-01'),
refreshedAt: null,
isVisible: true,
exclusionPatterns: [],
}),
externalLibraryWithExclusionPattern: Object.freeze<LibraryEntity>({
id: 'library-id',
name: 'test_library',
assets: [],
owner: userStub.externalPath1,
ownerId: 'user-id',
type: LibraryType.EXTERNAL,
importPaths: [],
createdAt: new Date('2023-01-01'),
updatedAt: new Date('2023-01-01'),
refreshedAt: null,
isVisible: true,
exclusionPatterns: ['**/dir1/**'],
}),
patternPath: Object.freeze<LibraryEntity>({
id: 'library-id1337',
name: 'importpath-exclusion-library1',
assets: [],
owner: userStub.externalPath1,
ownerId: 'user-id',
type: LibraryType.EXTERNAL,
importPaths: ['/xyz', '/asdf'],
createdAt: new Date('2023-01-01'),
updatedAt: new Date('2023-01-01'),
refreshedAt: null,
isVisible: true,
exclusionPatterns: ['**/dir1/**'],
}),
};

2
server/test/fixtures/system-config.stub.ts vendored
View File

@ -22,4 +22,6 @@ export const systemConfigStub: Record<string, SystemConfigEntity[]> = {
{ key: SystemConfigKey.OAUTH_MOBILE_REDIRECT_URI, value: 'http://mobile-redirect' },
{ key: SystemConfigKey.OAUTH_BUTTON_TEXT, value: 'OAuth' },
],
libraryWatchEnabled: [{ key: SystemConfigKey.LIBRARY_WATCH_ENABLED, value: true }],
libraryWatchDisabled: [{ key: SystemConfigKey.LIBRARY_WATCH_ENABLED, value: false }],
};

24
server/test/mocks/fswatcher.mock.ts Normal file
View File

@ -0,0 +1,24 @@
export const newFSWatcherMock = () => {
return {
options: {},
on: jest.fn(),
add: jest.fn(),
unwatch: jest.fn(),
getWatched: jest.fn(),
close: jest.fn(),
addListener: jest.fn(),
removeListener: jest.fn(),
removeAllListeners: jest.fn(),
eventNames: jest.fn(),
rawListeners: jest.fn(),
listeners: jest.fn(),
emit: jest.fn(),
listenerCount: jest.fn(),
off: jest.fn(),
once: jest.fn(),
prependListener: jest.fn(),
prependOnceListener: jest.fn(),
setMaxListeners: jest.fn(),
getMaxListeners: jest.fn(),
};
};

1
server/test/mocks/index.ts Normal file
View File

@ -0,0 +1 @@
export * from './fswatcher.mock';

1
server/test/repositories/storage.repository.mock.ts
View File

@ -21,5 +21,6 @@ export const newStorageRepositoryMock = (reset = true): jest.Mocked<IStorageRepo
crawl: jest.fn(),
rename: jest.fn(),
copyFile: jest.fn(),
watch: jest.fn(),
};
};

48
web/src/lib/components/admin-page/settings/library-settings/library-settings.svelte
View File

@ -26,13 +26,57 @@
<div>
<div in:fade={{ duration: 500 }}>
<SettingAccordion title="Scanning" subtitle="Settings for library scanning" isOpen>
<SettingAccordion title="Library watching (EXPERIMENTAL)" subtitle="Automatically watch for changed files" isOpen>
<form autocomplete="off" on:submit|preventDefault>
<div class="ml-4 mt-4 flex flex-col gap-4">
<SettingSwitch
title="Watch filesystem"
{disabled}
subtitle="Watch external libraries for file changes"
bind:checked={config.library.watch.enabled}
/>
<SettingSwitch
title="Use filesystem polling (EXPERIMENTAL)"
disabled={disabled || !config.library.watch.enabled}
subtitle="Use polling instead of native filesystem watching. This is required for network shares but can be very resource intensive. Use with care!"
bind:checked={config.library.watch.usePolling}
/>
<SettingInputField
inputType={SettingInputFieldType.NUMBER}
required={config.library.watch.usePolling}
disabled={disabled || !config.library.watch.usePolling || !config.library.watch.enabled}
label="Polling interval"
bind:value={config.library.watch.interval}
isEdited={config.library.watch.interval !== savedConfig.library.watch.interval}
>
<svelte:fragment slot="desc">
<p class="text-sm dark:text-immich-dark-fg">
Interval of filesystem polling, in milliseconds. Lower values will result in higher CPU usage.
</p>
</svelte:fragment>
</SettingInputField>
</div>
<div class="ml-4">
<SettingButtonsRow
on:reset={({ detail }) => dispatch('reset', { ...detail, configKeys: ['library'] })}
on:save={() => dispatch('save', { library: config.library })}
showResetToDefault={!isEqual(savedConfig.library, defaultConfig.library)}
{disabled}
/>
</div>
</form>
</SettingAccordion>
<SettingAccordion title="Periodic Scanning" subtitle="Configure periodic library scanning" isOpen>
<form autocomplete="off" on:submit|preventDefault>
<div class="ml-4 mt-4 flex flex-col gap-4">
<SettingSwitch
title="ENABLED"
{disabled}
subtitle="Enable automatic library scanning"
subtitle="Enable periodic library scanning"
bind:checked={config.library.scan.enabled}
/>