1
0
mirror of https://github.com/immich-app/immich.git synced 2024-11-24 08:52:28 +02:00

feat(server,web) Semantic import path validation (#7076)

* add library validation api

* chore: open api

* show warning i UI

* add flex row

* fix e2e

* tests

* fix tests

* enforce path validation

* enforce validation on refresh

* return 400 on bad import path

* add limits to import paths

* set response code to 200

* fix e2e

* fix lint

* fix test

* restore e2e folder

* fix import

* use startsWith

* icon color

* notify user of failed validation

* add parent div to validation

* add docs to the import validation

* improve library troubleshooting docs

* fix button alignment

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
This commit is contained in:
Jonathan Jogenfors 2024-02-20 16:53:12 +01:00 committed by GitHub
parent e7a875eadd
commit b3c7bebbd4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
32 changed files with 848 additions and 75 deletions

View File

@ -42,23 +42,24 @@ Finally, files can be deleted from Immich via the `Remove Offline Files` job. Th
### Import Paths
External libraries use import paths to determine which files to scan. Each library can have multiple import paths so that files from different locations can be added to the same library. Import paths are scanned recursively, and if a file is in multiple import paths, it will only be added once. If the import paths are edited in a way that an external file is no longer in any import path, it will be removed from the library in the same way a deleted file would. If the file is moved back to an import path, it will be added again as if it was a new file.
External libraries use import paths to determine which files to scan. Each library can have multiple import paths so that files from different locations can be added to the same library. Import paths are scanned recursively, and if a file is in multiple import paths, it will only be added once. Each import file must be a readable directory that exists on the filesystem; the import path dialog will alert you of any paths that are not accessible.
If the import paths are edited in a way that an external file is no longer in any import path, it will be removed from the library in the same way a deleted file would. If the file is moved back to an import path, it will be added again as if it was a new file.
### Troubleshooting
Sometimes, an external library will not scan correctly. This can happen if the immich_server or immich_microservices can't access the files. Here are some things to check:
Sometimes, an external library will not scan correctly. This can happen if immich_server or immich_microservices can't access the files. Here are some things to check:
- Is the external path set correctly?
- Is the external path set correctly? Each import path must be contained in the external path.
- Make sure the external path does not contain spaces
- In the docker-compose file, are the volumes mounted correctly?
- Are the volumes identical between the `server` and `microservices` container?
- Are the import paths set correctly, and do they match the path set in docker-compose file?
- Are you using symbolic link in your import library?
- Make sure you don't use symlinks in your import libraries, and that you aren't linking across docker mounts.
- Are the permissions set correctly?
- Are you using forward slashes everywhere? (`/`)
- Are you using symlink across docker mounts?
- Are you using [spaces in the internal path](/docs/features/libraries#:~:text=can%20be%20accessed.-,NOTE,-Spaces%20in%20the)?
- Make sure you are using forward slashes (`/`) and not backward slashes.
If all else fails, you can always start a shell inside the container and check if the path is accessible. For example, `docker exec -it immich_microservices /bin/bash` will start a bash shell. If your import path, for instance, is `/data/import/photos`, you can check if the files are accessible by running `ls /data/import/photos`. Also check the `immich_server` container in the same way.
To validate that Immich can reach your external library, start a shell inside the container. Run `docker exec -it immich_microservices /bin/bash` to a bash shell. If your import path is `/data/import/photos`, check it with `ls /data/import/photos`. Do the same check for the `immich_server` container. If you cannot access this directory in both the `microservices` and `server` containers, Immich won't be able to import files.
### Security Considerations

View File

@ -182,6 +182,9 @@ doc/UserAvatarColor.md
doc/UserDto.md
doc/UserResponseDto.md
doc/ValidateAccessTokenResponseDto.md
doc/ValidateLibraryDto.md
doc/ValidateLibraryImportPathResponseDto.md
doc/ValidateLibraryResponseDto.md
doc/VideoCodec.md
git_push.sh
lib/api.dart
@ -372,6 +375,9 @@ lib/model/user_avatar_color.dart
lib/model/user_dto.dart
lib/model/user_response_dto.dart
lib/model/validate_access_token_response_dto.dart
lib/model/validate_library_dto.dart
lib/model/validate_library_import_path_response_dto.dart
lib/model/validate_library_response_dto.dart
lib/model/video_codec.dart
pubspec.yaml
test/activity_api_test.dart
@ -553,4 +559,7 @@ test/user_avatar_color_test.dart
test/user_dto_test.dart
test/user_response_dto_test.dart
test/validate_access_token_response_dto_test.dart
test/validate_library_dto_test.dart
test/validate_library_import_path_response_dto_test.dart
test/validate_library_response_dto_test.dart
test/video_codec_test.dart

BIN
mobile/openapi/README.md generated

Binary file not shown.

Binary file not shown.

BIN
mobile/openapi/doc/ValidateLibraryDto.md generated Normal file

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -3618,6 +3618,58 @@
]
}
},
"/library/{id}/validate": {
"post": {
"operationId": "validate",
"parameters": [
{
"name": "id",
"required": true,
"in": "path",
"schema": {
"format": "uuid",
"type": "string"
}
}
],
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ValidateLibraryDto"
}
}
},
"required": true
},
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ValidateLibraryResponseDto"
}
}
},
"description": ""
}
},
"security": [
{
"bearer": []
},
{
"cookie": []
},
{
"api_key": []
}
],
"tags": [
"Library"
]
}
},
"/oauth/authorize": {
"post": {
"operationId": "startOAuth",
@ -10406,6 +10458,52 @@
],
"type": "object"
},
"ValidateLibraryDto": {
"properties": {
"exclusionPatterns": {
"items": {
"type": "string"
},
"type": "array"
},
"importPaths": {
"items": {
"type": "string"
},
"type": "array"
}
},
"type": "object"
},
"ValidateLibraryImportPathResponseDto": {
"properties": {
"importPath": {
"type": "string"
},
"isValid": {
"default": false,
"type": "boolean"
},
"message": {
"type": "string"
}
},
"required": [
"importPath"
],
"type": "object"
},
"ValidateLibraryResponseDto": {
"properties": {
"importPaths": {
"items": {
"$ref": "#/components/schemas/ValidateLibraryImportPathResponseDto"
},
"type": "array"
}
},
"type": "object"
},
"VideoCodec": {
"enum": [
"h264",

View File

@ -5316,6 +5316,63 @@ export interface ValidateAccessTokenResponseDto {
*/
'authStatus': boolean;
}
/**
*
* @export
* @interface ValidateLibraryDto
*/
export interface ValidateLibraryDto {
/**
*
* @type {Array<string>}
* @memberof ValidateLibraryDto
*/
'exclusionPatterns'?: Array<string>;
/**
*
* @type {Array<string>}
* @memberof ValidateLibraryDto
*/
'importPaths'?: Array<string>;
}
/**
*
* @export
* @interface ValidateLibraryImportPathResponseDto
*/
export interface ValidateLibraryImportPathResponseDto {
/**
*
* @type {string}
* @memberof ValidateLibraryImportPathResponseDto
*/
'importPath': string;
/**
*
* @type {boolean}
* @memberof ValidateLibraryImportPathResponseDto
*/
'isValid'?: boolean;
/**
*
* @type {string}
* @memberof ValidateLibraryImportPathResponseDto
*/
'message'?: string;
}
/**
*
* @export
* @interface ValidateLibraryResponseDto
*/
export interface ValidateLibraryResponseDto {
/**
*
* @type {Array<ValidateLibraryImportPathResponseDto>}
* @memberof ValidateLibraryResponseDto
*/
'importPaths'?: Array<ValidateLibraryImportPathResponseDto>;
}
/**
*
* @export
@ -12813,6 +12870,54 @@ export const LibraryApiAxiosParamCreator = function (configuration?: Configurati
localVarRequestOptions.headers = {...localVarHeaderParameter, ...headersFromBaseOptions, ...options.headers};
localVarRequestOptions.data = serializeDataIfNeeded(updateLibraryDto, localVarRequestOptions, configuration)
return {
url: toPathString(localVarUrlObj),
options: localVarRequestOptions,
};
},
/**
*
* @param {string} id
* @param {ValidateLibraryDto} validateLibraryDto
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
validate: async (id: string, validateLibraryDto: ValidateLibraryDto, options: RawAxiosRequestConfig = {}): Promise<RequestArgs> => {
// verify required parameter 'id' is not null or undefined
assertParamExists('validate', 'id', id)
// verify required parameter 'validateLibraryDto' is not null or undefined
assertParamExists('validate', 'validateLibraryDto', validateLibraryDto)
const localVarPath = `/library/{id}/validate`
.replace(`{${"id"}}`, encodeURIComponent(String(id)));
// use dummy base URL string because the URL constructor only accepts absolute URLs.
const localVarUrlObj = new URL(localVarPath, DUMMY_BASE_URL);
let baseOptions;
if (configuration) {
baseOptions = configuration.baseOptions;
}
const localVarRequestOptions = { method: 'POST', ...baseOptions, ...options};
const localVarHeaderParameter = {} as any;
const localVarQueryParameter = {} as any;
// authentication cookie required
// authentication api_key required
await setApiKeyToObject(localVarHeaderParameter, "x-api-key", configuration)
// authentication bearer required
// http bearer authentication required
await setBearerAuthToObject(localVarHeaderParameter, configuration)
localVarHeaderParameter['Content-Type'] = 'application/json';
setSearchParams(localVarUrlObj, localVarQueryParameter);
let headersFromBaseOptions = baseOptions && baseOptions.headers ? baseOptions.headers : {};
localVarRequestOptions.headers = {...localVarHeaderParameter, ...headersFromBaseOptions, ...options.headers};
localVarRequestOptions.data = serializeDataIfNeeded(validateLibraryDto, localVarRequestOptions, configuration)
return {
url: toPathString(localVarUrlObj),
options: localVarRequestOptions,
@ -12925,6 +13030,19 @@ export const LibraryApiFp = function(configuration?: Configuration) {
const operationBasePath = operationServerMap['LibraryApi.updateLibrary']?.[index]?.url;
return (axios, basePath) => createRequestFunction(localVarAxiosArgs, globalAxios, BASE_PATH, configuration)(axios, operationBasePath || basePath);
},
/**
*
* @param {string} id
* @param {ValidateLibraryDto} validateLibraryDto
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
async validate(id: string, validateLibraryDto: ValidateLibraryDto, options?: RawAxiosRequestConfig): Promise<(axios?: AxiosInstance, basePath?: string) => AxiosPromise<ValidateLibraryResponseDto>> {
const localVarAxiosArgs = await localVarAxiosParamCreator.validate(id, validateLibraryDto, options);
const index = configuration?.serverIndex ?? 0;
const operationBasePath = operationServerMap['LibraryApi.validate']?.[index]?.url;
return (axios, basePath) => createRequestFunction(localVarAxiosArgs, globalAxios, BASE_PATH, configuration)(axios, operationBasePath || basePath);
},
}
};
@ -13006,6 +13124,15 @@ export const LibraryApiFactory = function (configuration?: Configuration, basePa
updateLibrary(requestParameters: LibraryApiUpdateLibraryRequest, options?: RawAxiosRequestConfig): AxiosPromise<LibraryResponseDto> {
return localVarFp.updateLibrary(requestParameters.id, requestParameters.updateLibraryDto, options).then((request) => request(axios, basePath));
},
/**
*
* @param {LibraryApiValidateRequest} requestParameters Request parameters.
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
validate(requestParameters: LibraryApiValidateRequest, options?: RawAxiosRequestConfig): AxiosPromise<ValidateLibraryResponseDto> {
return localVarFp.validate(requestParameters.id, requestParameters.validateLibraryDto, options).then((request) => request(axios, basePath));
},
};
};
@ -13121,6 +13248,27 @@ export interface LibraryApiUpdateLibraryRequest {
readonly updateLibraryDto: UpdateLibraryDto
}
/**
* Request parameters for validate operation in LibraryApi.
* @export
* @interface LibraryApiValidateRequest
*/
export interface LibraryApiValidateRequest {
/**
*
* @type {string}
* @memberof LibraryApiValidate
*/
readonly id: string
/**
*
* @type {ValidateLibraryDto}
* @memberof LibraryApiValidate
*/
readonly validateLibraryDto: ValidateLibraryDto
}
/**
* LibraryApi - object-oriented interface
* @export
@ -13214,6 +13362,17 @@ export class LibraryApi extends BaseAPI {
public updateLibrary(requestParameters: LibraryApiUpdateLibraryRequest, options?: RawAxiosRequestConfig) {
return LibraryApiFp(this.configuration).updateLibrary(requestParameters.id, requestParameters.updateLibraryDto, options).then((request) => request(this.axios, this.basePath));
}
/**
*
* @param {LibraryApiValidateRequest} requestParameters Request parameters.
* @param {*} [options] Override http request option.
* @throws {RequiredError}
* @memberof LibraryApi
*/
public validate(requestParameters: LibraryApiValidateRequest, options?: RawAxiosRequestConfig) {
return LibraryApiFp(this.configuration).validate(requestParameters.id, requestParameters.validateLibraryDto, options).then((request) => request(this.axios, this.basePath));
}
}

Binary file not shown.

View File

@ -1,6 +1,16 @@
import { PostgreSqlContainer } from '@testcontainers/postgresql';
import path from 'node:path';
export default async () => {
let IMMICH_TEST_ASSET_PATH: string = '';
if (process.env.IMMICH_TEST_ASSET_PATH === undefined) {
IMMICH_TEST_ASSET_PATH = path.normalize(`${__dirname}/../../test/assets/`);
process.env.IMMICH_TEST_ASSET_PATH = IMMICH_TEST_ASSET_PATH;
} else {
IMMICH_TEST_ASSET_PATH = process.env.IMMICH_TEST_ASSET_PATH;
}
const pg = await new PostgreSqlContainer('tensorchord/pgvecto-rs:pg14-v0.2.0')
.withDatabase('immich')
.withUsername('postgres')
@ -11,6 +21,9 @@ export default async () => {
process.env.DB_URL = pg.getConnectionUri();
process.env.NODE_ENV = 'development';
process.env.LOG_LEVEL = 'fatal';
process.env.TZ = 'Z';
if (process.env.LOG_LEVEL === undefined) {
process.env.LOG_LEVEL = 'fatal';
}
};

View File

@ -2,6 +2,7 @@ import { LibraryResponseDto, LoginResponseDto } from '@app/domain';
import { LibraryController } from '@app/immich';
import { LibraryType } from '@app/infra/entities';
import { errorStub, userDto, uuidStub } from '@test/fixtures';
import { IMMICH_TEST_ASSET_TEMP_PATH, restoreTempFolder } from 'src/test-utils/utils';
import request from 'supertest';
import { api } from '../../client';
import { testApp } from '../utils';
@ -20,6 +21,7 @@ describe(`${LibraryController.name} (e2e)`, () => {
});
beforeEach(async () => {
await restoreTempFolder();
await testApp.reset();
await api.authApi.adminSignUp(server);
admin = await api.authApi.adminLogin(server);
@ -247,15 +249,16 @@ describe(`${LibraryController.name} (e2e)`, () => {
});
it('should change the import paths', async () => {
await api.userApi.setExternalPath(server, admin.accessToken, admin.userId, IMMICH_TEST_ASSET_TEMP_PATH);
const { status, body } = await request(server)
.put(`/library/${library.id}`)
.set('Authorization', `Bearer ${admin.accessToken}`)
.send({ importPaths: ['/path/to/import'] });
.send({ importPaths: [IMMICH_TEST_ASSET_TEMP_PATH] });
expect(status).toBe(200);
expect(body).toEqual(
expect.objectContaining({
importPaths: ['/path/to/import'],
importPaths: [IMMICH_TEST_ASSET_TEMP_PATH],
}),
);
});
@ -435,4 +438,93 @@ describe(`${LibraryController.name} (e2e)`, () => {
expect(body).toEqual(errorStub.unauthorized);
});
});
describe('POST /library/:id/validate', () => {
it('should require authentication', async () => {
const { status, body } = await request(server).post(`/library/${uuidStub.notFound}/validate`).send({});
expect(status).toBe(401);
expect(body).toEqual(errorStub.unauthorized);
});
describe('Validate import path', () => {
let library: LibraryResponseDto;
beforeEach(async () => {
// Create an external library with default settings
library = await api.libraryApi.create(server, admin.accessToken, { type: LibraryType.EXTERNAL });
});
it('should fail with no external path set', async () => {
const { status, body } = await request(server)
.post(`/library/${library.id}/validate`)
.set('Authorization', `Bearer ${admin.accessToken}`)
.send({ importPaths: [] });
expect(status).toBe(400);
expect(body).toEqual(errorStub.badRequest('User has no external path set'));
});
describe('With external path set', () => {
beforeEach(async () => {
await api.userApi.setExternalPath(server, admin.accessToken, admin.userId, IMMICH_TEST_ASSET_TEMP_PATH);
});
it('should pass with no import paths', async () => {
const response = await api.libraryApi.validate(server, admin.accessToken, library.id, { importPaths: [] });
expect(response.importPaths).toEqual([]);
});
it('should not allow paths outside of the external path', async () => {
const pathToTest = `${IMMICH_TEST_ASSET_TEMP_PATH}/../`;
const response = await api.libraryApi.validate(server, admin.accessToken, library.id, {
importPaths: [pathToTest],
});
expect(response.importPaths?.length).toEqual(1);
const pathResponse = response?.importPaths?.at(0);
expect(pathResponse).toEqual({
importPath: pathToTest,
isValid: false,
message: `Not contained in user's external path`,
});
});
it('should fail if path does not exist', async () => {
const pathToTest = `${IMMICH_TEST_ASSET_TEMP_PATH}/does/not/exist`;
const response = await api.libraryApi.validate(server, admin.accessToken, library.id, {
importPaths: [pathToTest],
});
expect(response.importPaths?.length).toEqual(1);
const pathResponse = response?.importPaths?.at(0);
expect(pathResponse).toEqual({
importPath: pathToTest,
isValid: false,
message: `Path does not exist (ENOENT)`,
});
});
it('should fail if path is a file', async () => {
const pathToTest = `${IMMICH_TEST_ASSET_TEMP_PATH}/does/not/exist`;
const response = await api.libraryApi.validate(server, admin.accessToken, library.id, {
importPaths: [pathToTest],
});
expect(response.importPaths?.length).toEqual(1);
const pathResponse = response?.importPaths?.at(0);
expect(pathResponse).toEqual({
importPath: pathToTest,
isValid: false,
message: `Path does not exist (ENOENT)`,
});
});
});
});
});
});

View File

@ -4,6 +4,8 @@ import {
LibraryStatsResponseDto,
ScanLibraryDto,
UpdateLibraryDto,
ValidateLibraryDto,
ValidateLibraryResponseDto,
} from '@app/domain';
import request from 'supertest';
@ -58,4 +60,12 @@ export const libraryApi = {
expect(status).toBe(200);
return body as LibraryResponseDto;
},
validate: async (server: any, accessToken: string, id: string, data: ValidateLibraryDto) => {
const { body, status } = await request(server)
.post(`/library/${id}/validate`)
.set('Authorization', `Bearer ${accessToken}`)
.send(data);
expect(status).toBe(200);
return body as ValidateLibraryResponseDto;
},
};

View File

@ -1,6 +1,6 @@
import { LibraryEntity, LibraryType } from '@app/infra/entities';
import { ApiProperty } from '@nestjs/swagger';
import { ArrayUnique, IsBoolean, IsEnum, IsNotEmpty, IsOptional, IsString } from 'class-validator';
import { ArrayMaxSize, ArrayUnique, IsBoolean, IsEnum, IsNotEmpty, IsOptional, IsString } from 'class-validator';
import { ValidateUUID } from '../domain.util';
export class CreateLibraryDto {
@ -21,12 +21,14 @@ export class CreateLibraryDto {
@IsString({ each: true })
@IsNotEmpty({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
importPaths?: string[];
@IsOptional()
@IsString({ each: true })
@IsNotEmpty({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
exclusionPatterns?: string[];
@IsOptional()
@ -48,12 +50,14 @@ export class UpdateLibraryDto {
@IsString({ each: true })
@IsNotEmpty({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
importPaths?: string[];
@IsOptional()
@IsNotEmpty({ each: true })
@IsString({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
exclusionPatterns?: string[];
}
@ -63,6 +67,32 @@ export class CrawlOptionsDto {
exclusionPatterns?: string[];
}
export class ValidateLibraryDto {
@IsOptional()
@IsString({ each: true })
@IsNotEmpty({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
importPaths?: string[];
@IsOptional()
@IsNotEmpty({ each: true })
@IsString({ each: true })
@ArrayUnique()
@ArrayMaxSize(128)
exclusionPatterns?: string[];
}
export class ValidateLibraryResponseDto {
importPaths?: ValidateLibraryImportPathResponseDto[];
}
export class ValidateLibraryImportPathResponseDto {
importPath!: string;
isValid?: boolean = false;
message?: string;
}
export class LibrarySearchDto {
@ValidateUUID({ optional: true })
userId?: string;

View File

@ -54,12 +54,6 @@ describe(LibraryService.name, () => {
cryptoMock = newCryptoRepositoryMock();
storageMock = newStorageRepositoryMock();
storageMock.stat.mockResolvedValue({
size: 100,
mtime: new Date('2023-01-01'),
ctime: new Date('2023-01-01'),
} as Stats);
// Always validate owner access for library.
accessMock.library.checkOwnerAccess.mockImplementation(async (_, libraryIds) => libraryIds);
@ -270,6 +264,39 @@ describe(LibraryService.name, () => {
await expect(sut.handleQueueAssetRefresh(mockLibraryJob)).resolves.toBe(false);
});
it('should ignore import paths that do not exist', async () => {
storageMock.stat.mockImplementation((path): Promise<Stats> => {
if (path === libraryStub.externalLibraryWithImportPaths1.importPaths[0]) {
const error = { code: 'ENOENT' } as any;
throw error;
}
return Promise.resolve({
isDirectory: () => true,
} as Stats);
});
storageMock.checkFileExists.mockResolvedValue(true);
const mockLibraryJob: ILibraryRefreshJob = {
id: libraryStub.externalLibraryWithImportPaths1.id,
refreshModifiedFiles: false,
refreshAllFiles: false,
};
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
storageMock.crawl.mockResolvedValue([]);
assetMock.getByLibraryId.mockResolvedValue([]);
libraryMock.getOnlineAssetPaths.mockResolvedValue([]);
userMock.get.mockResolvedValue(userStub.externalPathRoot);
await sut.handleQueueAssetRefresh(mockLibraryJob);
expect(storageMock.crawl).toHaveBeenCalledWith({
pathsToCrawl: [libraryStub.externalLibraryWithImportPaths1.importPaths[1]],
exclusionPatterns: [],
});
});
});
describe('handleAssetRefresh', () => {
@ -278,6 +305,12 @@ describe(LibraryService.name, () => {
beforeEach(() => {
mockUser = userStub.externalPath1;
userMock.get.mockResolvedValue(mockUser);
storageMock.stat.mockResolvedValue({
size: 100,
mtime: new Date('2023-01-01'),
ctime: new Date('2023-01-01'),
} as Stats);
});
it('should reject an unknown file extension', async () => {
@ -1104,13 +1137,19 @@ describe(LibraryService.name, () => {
libraryMock.update.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1);
await expect(sut.update(authStub.admin, authStub.admin.user.id, { importPaths: ['/foo'] })).resolves.toEqual(
mapLibrary(libraryStub.externalLibraryWithImportPaths1),
);
storageMock.stat.mockResolvedValue({
isDirectory: () => true,
} as Stats);
storageMock.checkFileExists.mockResolvedValue(true);
await expect(
sut.update(authStub.external1, authStub.external1.user.id, { importPaths: ['/data/user1/foo'] }),
).resolves.toEqual(mapLibrary(libraryStub.externalLibraryWithImportPaths1));
expect(libraryMock.update).toHaveBeenCalledWith(
expect.objectContaining({
id: authStub.admin.user.id,
id: authStub.external1.user.id,
}),
);
expect(storageMock.watch).toHaveBeenCalledWith(
@ -1142,7 +1181,7 @@ describe(LibraryService.name, () => {
});
});
describe('watchAll new', () => {
describe('watchAll', () => {
describe('watching disabled', () => {
beforeEach(async () => {
configMock.load.mockResolvedValue(systemConfigStub.libraryWatchDisabled);
@ -1523,4 +1562,121 @@ describe(LibraryService.name, () => {
]);
});
});
describe('validate', () => {
it('should validate directory', async () => {
storageMock.stat.mockResolvedValue({
isDirectory: () => true,
} as Stats);
storageMock.checkFileExists.mockResolvedValue(true);
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user1/'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user1/',
isValid: true,
message: undefined,
},
]);
});
it('should error when no external path is set', async () => {
await expect(
sut.validate(authStub.admin, libraryStub.externalLibrary1.id, { importPaths: ['/photos'] }),
).rejects.toBeInstanceOf(BadRequestException);
});
it('should detect when path is outside external path', async () => {
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user2'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user2',
isValid: false,
message: "Not contained in user's external path",
},
]);
});
it('should detect when path does not exist', async () => {
storageMock.stat.mockImplementation(() => {
const error = { code: 'ENOENT' } as any;
throw error;
});
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user1/'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user1/',
isValid: false,
message: 'Path does not exist (ENOENT)',
},
]);
});
it('should detect when path is not a directory', async () => {
storageMock.stat.mockResolvedValue({
isDirectory: () => false,
} as Stats);
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user1/file'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user1/file',
isValid: false,
message: 'Not a directory',
},
]);
});
it('should return an unknown exception from stat', async () => {
storageMock.stat.mockImplementation(() => {
throw new Error('Unknown error');
});
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user1/'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user1/',
isValid: false,
message: 'Error: Unknown error',
},
]);
});
it('should detect when access rights are missing', async () => {
storageMock.stat.mockResolvedValue({
isDirectory: () => true,
} as Stats);
storageMock.checkFileExists.mockResolvedValue(false);
const result = await sut.validate(authStub.external1, libraryStub.externalLibraryWithImportPaths1.id, {
importPaths: ['/data/user1/'],
});
expect(result.importPaths).toEqual([
{
importPath: '/data/user1/',
isValid: false,
message: 'Lacking read permission for folder',
},
]);
});
});
});

View File

@ -30,6 +30,9 @@ import {
LibraryStatsResponseDto,
ScanLibraryDto,
UpdateLibraryDto,
ValidateLibraryDto,
ValidateLibraryImportPathResponseDto,
ValidateLibraryResponseDto,
mapLibrary,
} from './library.dto';
@ -270,10 +273,81 @@ export class LibraryService extends EventEmitter {
);
}
private async validateImportPath(importPath: string): Promise<ValidateLibraryImportPathResponseDto> {
const validation = new ValidateLibraryImportPathResponseDto();
validation.importPath = importPath;
try {
const stat = await this.storageRepository.stat(importPath);
if (!stat.isDirectory()) {
validation.message = 'Not a directory';
return validation;
}
} catch (error: any) {
if (error.code === 'ENOENT') {
validation.message = 'Path does not exist (ENOENT)';
return validation;
}
validation.message = String(error);
return validation;
}
const access = await this.storageRepository.checkFileExists(importPath, R_OK);
if (!access) {
validation.message = 'Lacking read permission for folder';
return validation;
}
validation.isValid = true;
return validation;
}
public async validate(auth: AuthDto, id: string, dto: ValidateLibraryDto): Promise<ValidateLibraryResponseDto> {
await this.access.requirePermission(auth, Permission.LIBRARY_UPDATE, id);
if (!auth.user.externalPath) {
throw new BadRequestException('User has no external path set');
}
const response = new ValidateLibraryResponseDto();
if (dto.importPaths) {
response.importPaths = await Promise.all(
dto.importPaths.map(async (importPath) => {
const normalizedPath = path.normalize(importPath);
if (!this.isInExternalPath(normalizedPath, auth.user.externalPath)) {
const validation = new ValidateLibraryImportPathResponseDto();
validation.importPath = importPath;
validation.message = `Not contained in user's external path`;
return validation;
}
return await this.validateImportPath(importPath);
}),
);
}
return response;
}
async update(auth: AuthDto, id: string, dto: UpdateLibraryDto): Promise<LibraryResponseDto> {
await this.access.requirePermission(auth, Permission.LIBRARY_UPDATE, id);
const library = await this.repository.update({ id, ...dto });
if (dto.importPaths) {
const validation = await this.validate(auth, id, { importPaths: dto.importPaths });
if (validation.importPaths) {
for (const path of validation.importPaths) {
if (!path.isValid) {
throw new BadRequestException(`Invalid import path: ${path.message}`);
}
}
}
}
if (dto.importPaths || dto.exclusionPatterns) {
// Re-watch library to use new paths and/or exclusion patterns
await this.watch(id);
@ -509,6 +583,14 @@ export class LibraryService extends EventEmitter {
return true;
}
// Check if a given path is in a user's external path. Both arguments are assumed to be normalized
private isInExternalPath(filePath: string, externalPath: string | null): boolean {
if (externalPath === null) {
return false;
}
return filePath.startsWith(externalPath);
}
async handleQueueAssetRefresh(job: ILibraryRefreshJob): Promise<boolean> {
const library = await this.repository.get(job.id);
if (!library || library.type !== LibraryType.EXTERNAL) {
@ -523,17 +605,31 @@ export class LibraryService extends EventEmitter {
}
this.logger.verbose(`Refreshing library: ${job.id}`);
const pathValidation = await Promise.all(
library.importPaths.map(async (importPath) => await this.validateImportPath(importPath)),
);
const validImportPaths = pathValidation
.map((validation) => {
if (!validation.isValid) {
this.logger.error(`Skipping invalid import path: ${validation.importPath}. Reason: ${validation.message}`);
}
return validation;
})
.filter((validation) => validation.isValid)
.map((validation) => validation.importPath);
const rawPaths = await this.storageRepository.crawl({
pathsToCrawl: library.importPaths,
pathsToCrawl: validImportPaths,
exclusionPatterns: library.exclusionPatterns,
});
const crawledAssetPaths = rawPaths
// Normalize file paths. This is important to prevent security issues like path traversal
.map((filePath) => path.normalize(filePath))
.filter((assetPath) =>
// Filter out paths that are not within the user's external path
assetPath.match(new RegExp(`^${user.externalPath}`)),
) as string[];
// Filter out paths that are not within the user's external path
.filter((assetPath) => this.isInExternalPath(assetPath, user.externalPath)) as string[];
this.logger.debug(`Found ${crawledAssetPaths.length} asset(s) when crawling import paths ${library.importPaths}`);
const assetsInLibrary = await this.assetRepository.getByLibraryId([job.id]);

View File

@ -6,8 +6,10 @@ import {
LibraryResponseDto as ResponseDto,
ScanLibraryDto,
UpdateLibraryDto as UpdateDto,
ValidateLibraryDto,
ValidateLibraryResponseDto,
} from '@app/domain';
import { Body, Controller, Delete, Get, Param, Post, Put } from '@nestjs/common';
import { Body, Controller, Delete, Get, HttpCode, Param, Post, Put } from '@nestjs/common';
import { ApiTags } from '@nestjs/swagger';
import { Auth, Authenticated } from '../app.guard';
import { UseValidation } from '../app.utils';
@ -40,6 +42,16 @@ export class LibraryController {
return this.service.get(auth, id);
}
@Post(':id/validate')
@HttpCode(200)
validate(
@Auth() auth: AuthDto,
@Param() { id }: UUIDParamDto,
@Body() dto: ValidateLibraryDto,
): Promise<ValidateLibraryResponseDto> {
return this.service.validate(auth, id, dto);
}
@Delete(':id')
deleteLibrary(@Auth() auth: AuthDto, @Param() { id }: UUIDParamDto): Promise<void> {
return this.service.delete(auth, id);

View File

@ -164,7 +164,21 @@ export function searchAssetBuilder(
builder.andWhere(_.omitBy(path, _.isUndefined));
const status = _.pick(options, ['isExternal', 'isFavorite', 'isOffline', 'isReadOnly', 'isVisible', 'type']);
const { isArchived, isEncoded, isMotion, withArchived } = options;
const {
isArchived,
isEncoded,
isMotion,
withArchived,
isNotInAlbum,
withFaces,
withPeople,
withSmartInfo,
personIds,
withExif,
withStacked,
trashedAfter,
trashedBefore,
} = options;
builder.andWhere(
_.omitBy(
{
@ -177,38 +191,38 @@ export function searchAssetBuilder(
),
);
if (options.isNotInAlbum) {
if (isNotInAlbum) {
builder
.leftJoin(`${builder.alias}.albums`, 'albums')
.andWhere('albums.id IS NULL')
.andWhere(`${builder.alias}.isVisible = true`);
}
if (options.withFaces || options.withPeople) {
if (withFaces || withPeople) {
builder.leftJoinAndSelect(`${builder.alias}.faces`, 'faces');
}
if (options.withPeople) {
if (withPeople) {
builder.leftJoinAndSelect(`${builder.alias}.person`, 'person');
}
if (options.withSmartInfo) {
if (withSmartInfo) {
builder.leftJoinAndSelect(`${builder.alias}.smartInfo`, 'smartInfo');
}
if (options.personIds && options.personIds.length > 0) {
if (personIds && personIds.length > 0) {
builder
.leftJoin(`${builder.alias}.faces`, 'faces')
.andWhere('faces.personId IN (:...personIds)', { personIds: options.personIds })
.andWhere('faces.personId IN (:...personIds)', { personIds: personIds })
.addGroupBy(`${builder.alias}.id`)
.having('COUNT(faces.id) = :personCount', { personCount: options.personIds.length });
.having('COUNT(faces.id) = :personCount', { personCount: personIds.length });
if (options.withExif) {
if (withExif) {
builder.addGroupBy('exifInfo.assetId');
}
}
if (options.withStacked) {
if (withStacked) {
builder
.leftJoinAndSelect(`${builder.alias}.stack`, 'stack')
.leftJoinAndSelect('stack.assets', 'stackedAssets')
@ -217,8 +231,7 @@ export function searchAssetBuilder(
);
}
const withDeleted =
options.withDeleted ?? (options.trashedAfter !== undefined || options.trashedBefore !== undefined);
const withDeleted = options.withDeleted ?? (trashedAfter !== undefined || trashedBefore !== undefined);
if (withDeleted) {
builder.withDeleted();
}

View File

@ -12,12 +12,24 @@ import archiver from 'archiver';
import chokidar, { WatchOptions } from 'chokidar';
import { glob } from 'glob';
import { constants, createReadStream, existsSync, mkdirSync } from 'node:fs';
import fs, { copyFile, readdir, rename, utimes, writeFile } from 'node:fs/promises';
import fs, { copyFile, readdir, rename, stat, utimes, writeFile } from 'node:fs/promises';
import path from 'node:path';
export class FilesystemProvider implements IStorageRepository {
private logger = new ImmichLogger(FilesystemProvider.name);
readdir = readdir;
copyFile = copyFile;
stat = stat;
writeFile = writeFile;
rename = rename;
utimes = utimes;
createZipStream(): ImmichZipStream {
const archive = archiver('zip', { store: true });
@ -50,14 +62,6 @@ export class FilesystemProvider implements IStorageRepository {
}
}
writeFile = writeFile;
rename = rename;
copyFile = copyFile;
utimes = utimes;
async checkFileExists(filepath: string, mode = constants.F_OK): Promise<boolean> {
try {
await fs.access(filepath, mode);
@ -79,8 +83,6 @@ export class FilesystemProvider implements IStorageRepository {
}
}
stat = fs.stat;
async unlinkDir(folder: string, options: { recursive?: boolean; force?: boolean }) {
await fs.rm(folder, options);
}
@ -146,6 +148,4 @@ export class FilesystemProvider implements IStorageRepository {
return () => watcher.close();
}
readdir = readdir;
}

View File

@ -140,6 +140,25 @@ export const userStub = {
quotaSizeInBytes: null,
quotaUsageInBytes: 0,
}),
externalPathRoot: Object.freeze<UserEntity>({
...authStub.user1.user,
password: 'immich_password',
name: 'immich_name',
storageLabel: 'label-1',
externalPath: '/',
oauthId: '',
shouldChangePassword: false,
profileImagePath: '',
createdAt: new Date('2021-01-01'),
deletedAt: null,
updatedAt: new Date('2021-01-01'),
tags: [],
assets: [],
memoriesEnabled: true,
avatarColor: UserAvatarColor.PRIMARY,
quotaSizeInBytes: null,
quotaUsageInBytes: 0,
}),
profilePath: Object.freeze<UserEntity>({
...authStub.user1.user,
password: 'immich_password',

View File

@ -1,13 +1,15 @@
<script lang="ts">
import Icon from '$lib/components/elements/icon.svelte';
import type { LibraryResponseDto } from '@immich/sdk';
import { mdiPencilOutline } from '@mdi/js';
import { createEventDispatcher, onMount } from 'svelte';
import { handleError } from '../../utils/handle-error';
import Button from '../elements/buttons/button.svelte';
import LibraryImportPathForm from './library-import-path-form.svelte';
import Icon from '$lib/components/elements/icon.svelte';
import { mdiAlertOutline, mdiCheckCircleOutline, mdiPencilOutline, mdiRefresh } from '@mdi/js';
import { validate, type LibraryResponseDto } from '@immich/sdk';
import type { ValidateLibraryImportPathResponseDto } from '@immich/sdk/axios';
import { NotificationType, notificationController } from '../shared-components/notification/notification';
export let library: Partial<LibraryResponseDto>;
export let library: LibraryResponseDto;
let addImportPath = false;
let editImportPath: number | null = null;
@ -15,20 +17,62 @@
let importPathToAdd: string | null = null;
let editedImportPath: string;
let importPaths: string[] = [];
let validatedPaths: ValidateLibraryImportPathResponseDto[] = [];
onMount(() => {
$: importPaths = validatedPaths.map((validatedPath) => validatedPath.importPath);
onMount(async () => {
if (library.importPaths) {
importPaths = library.importPaths;
await handleValidation();
} else {
library.importPaths = [];
}
});
const handleValidation = async () => {
if (library.importPaths) {
const validation = await validate({
id: library.id,
validateLibraryDto: { importPaths: library.importPaths },
});
validatedPaths = validation.importPaths ?? [];
}
};
const revalidate = async (notifyIfSuccessful = true) => {
await handleValidation();
let failedPaths = 0;
for (const validatedPath of validatedPaths) {
if (!validatedPath.isValid) {
failedPaths++;
}
}
if (failedPaths === 0) {
if (notifyIfSuccessful) {
notificationController.show({
message: `All paths validated successfully`,
type: NotificationType.Info,
});
}
} else if (failedPaths === 1) {
notificationController.show({
message: `${failedPaths} path failed validation`,
type: NotificationType.Warning,
});
} else {
notificationController.show({
message: `${failedPaths} paths failed validation`,
type: NotificationType.Warning,
});
}
};
const dispatch = createEventDispatcher<{
cancel: void;
submit: Partial<LibraryResponseDto>;
}>();
const handleCancel = () => {
dispatch('cancel');
};
@ -50,7 +94,7 @@
// Check so that import path isn't duplicated
if (!library.importPaths.includes(importPathToAdd)) {
library.importPaths.push(importPathToAdd);
importPaths = library.importPaths;
await revalidate(false);
}
} catch (error) {
handleError(error, 'Unable to add import path');
@ -75,7 +119,7 @@
if (!library.importPaths.includes(editedImportPath)) {
// Update import path
library.importPaths[editImportPath] = editedImportPath;
importPaths = library.importPaths;
await revalidate(false);
}
} catch (error) {
editImportPath = null;
@ -97,7 +141,7 @@
const pathToDelete = library.importPaths[editImportPath];
library.importPaths = library.importPaths.filter((path) => path != pathToDelete);
importPaths = library.importPaths;
await handleValidation();
} catch (error) {
handleError(error, 'Unable to delete import path');
} finally {
@ -138,7 +182,7 @@
<form on:submit|preventDefault={() => handleSubmit()} autocomplete="off" class="m-4 flex flex-col gap-4">
<table class="text-left">
<tbody class="block w-full overflow-y-auto rounded-md border dark:border-immich-dark-gray">
{#each importPaths as importPath, listIndex}
{#each validatedPaths as validatedPath, listIndex}
<tr
class={`flex h-[80px] w-full place-items-center text-center dark:text-immich-dark-fg ${
listIndex % 2 == 0
@ -146,13 +190,31 @@
: 'bg-immich-bg dark:bg-immich-dark-gray/50'
}`}
>
<td class="w-4/5 text-ellipsis px-4 text-sm">{importPath}</td>
<td class="w-1/5 text-ellipsis px-4 text-sm">
<td class="w-1/8 text-ellipsis pl-8 text-sm">
{#if validatedPath.isValid}
<Icon
path={mdiCheckCircleOutline}
size="24"
title={validatedPath.message}
class="text-immich-success dark:text-immich-dark-success"
/>
{:else}
<Icon
path={mdiAlertOutline}
size="24"
title={validatedPath.message}
class="text-immich-warning dark:text-immich-dark-warning"
/>
{/if}
</td>
<td class="w-4/5 text-ellipsis px-4 text-sm">{validatedPath.importPath}</td>
<td class="w-1/5 text-ellipsis px-4 text-sm flex flex-row">
<button
type="button"
on:click={() => {
editImportPath = listIndex;
editedImportPath = importPath;
editedImportPath = validatedPath.importPath;
}}
class="rounded-full bg-immich-primary p-3 text-gray-100 transition-all duration-150 hover:bg-immich-primary/75 dark:bg-immich-dark-primary dark:text-gray-700"
>
@ -185,9 +247,13 @@
>
</tbody>
</table>
<div class="flex w-full justify-end gap-2">
<Button size="sm" color="gray" on:click={() => handleCancel()}>Cancel</Button>
<Button size="sm" type="submit">Save</Button>
<div class="flex justify-between w-full">
<div class="justify-end gap-2">
<Button size="sm" color="gray" on:click={() => revalidate()}><Icon path={mdiRefresh} size={20} />Validate</Button>
</div>
<div class="justify-end gap-2">
<Button size="sm" color="gray" on:click={() => handleCancel()}>Cancel</Button>
<Button size="sm" type="submit">Save</Button>
</div>
</div>
</form>

View File

@ -126,11 +126,10 @@
try {
const libraryId = libraries[updateLibraryIndex].id;
await updateLibrary({ id: libraryId, updateLibraryDto: { ...event } });
} catch (error) {
handleError(error, 'Unable to update library');
} finally {
closeAll();
await readLibraryList();
} catch (error) {
handleError(error, 'Unable to update library');
}
};