feat(ml): support multiple urls (#14347)

* support multiple url

* update api

* styling

unnecessary `?.`

* update docs, make new url field go first

add load balancing section

* update tests

doc formatting

wording

wording

linting

* small styling

* `url` -> `urls`

* fix tests

* update docs

* make docusaurus happy

---------

Co-authored-by: Alex <alex.tran1502@gmail.com>

docs/docs/guides/remote-machine-learning.md

@@ -1,18 +1,20 @@
 # Remote Machine Learning
 
-To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-immich-slow-on-low-memory-systems-like-the-raspberry-pi) like the Raspberry Pi, you may also host Immich's machine-learning container on a more powerful system (e.g. your laptop or desktop computer):
-
-- Set the URL in Machine Learning Settings on the Admin Settings page to point to the designated ML system, e.g. `http://workstation:3003`.
-- Copy the following `docker-compose.yml` to your ML system.
-  - If using [hardware acceleration](/docs/features/ml-hardware-acceleration), the [hwaccel.ml.yml](https://github.com/immich-app/immich/releases/latest/download/hwaccel.ml.yml) file also needs to be added
-- Start the container by running `docker compose up -d`.
+To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-immich-slow-on-low-memory-systems-like-the-raspberry-pi) like the Raspberry Pi, you may also host Immich's machine learning container on a more powerful system, such as your laptop or desktop computer. The server container will send requests containing the image preview to the remote machine learning container for processing. The machine learning container does not persist this data or associate it with a particular user.
 
 :::info
-Smart Search and Face Detection will use this feature, but Facial Recognition is handled in the server.
+Smart Search and Face Detection will use this feature, but Facial Recognition will not. This is because Facial Recognition uses the _outputs_ of these models that have already been saved to the database. As such, its processing is between the server container and the database.
 :::
 
 :::danger
-When using remote machine learning, the thumbnails are sent to the remote machine learning container. Use this option carefully when running this on a public computer or a paid processing cloud.
+Image previews are sent to the remote machine learning container. Use this option carefully when running this on a public computer or a paid processing cloud. Additionally, as an internal service, the machine learning container has no security measures whatsoever. Please be mindful of where it's deployed and who can access it.
+:::
+
+1. Ensure the remote server has Docker installed
+2. Copy the following `docker-compose.yml` to the remote server
+
+:::info
+If using hardware acceleration, the [hwaccel.ml.yml](https://github.com/immich-app/immich/releases/latest/download/hwaccel.ml.yml) file also needs to be added and the `docker-compose.yml` needs to be configured as described in the [hardware acceleration documentation](/docs/features/ml-hardware-acceleration)
 :::
 
 ```yaml
@@ -37,8 +39,26 @@ volumes:
   model-cache:
 ```
 
-Please note that version mismatches between both hosts may cause instabilities and bugs, so make sure to always perform updates together.
+3. Start the remote machine learning container by running `docker compose up -d`
 
-:::caution
-As an internal service, the machine learning container has no security measures whatsoever. Please be mindful of where it's deployed and who can access it.
+:::info
+Version mismatches between both hosts may cause bugs and instability, so remember to update this container as well when updating the local Immich instance.
+:::
+
+4. Navigate to the [Machine Learning Settings](https://my.immich.app/admin/system-settings?isOpen=machine-learning)
+5. Click _Add URL_
+6. Fill the new field with the URL to the remote machine learning container, e.g. `http://ip:port`
+
+## Forcing remote processing
+
+Adding a new URL to the settings is recommended over replacing the existing URL. This is because it will allow machine learning tasks to be processed successfully when the remote server is down by falling back to the local machine learning container. If you do not want machine learning tasks to be processed locally when the remote server is not available, you can instead replace the existing URL and only provide the remote container's URL. If doing this, you can remove the `immich-machine-learning` section of the local `docker-compose.yml` file to save resources, as this service will never be used.
+
+Do note that this will mean that Smart Search and Face Detection jobs will fail to be processed when the remote instance is not available. This in turn means that tasks dependent on these features—Duplicate Detection and Facial Recognition—will not run for affected assets. If this occurs, you must manually click the _Missing_ button next to Smart Search and Face Detection in the [Job Status](http://my.immich.app/admin/jobs-status) page for the jobs to be retried.
+
+## Load balancing
+
+While several URLs can be provided in the settings, they are tried sequentially; there is no attempt to distribute load across multiple containers. It is recommended to use a dedicated load balancer for such use-cases and specify it as the only URL. Among other things, it may enable the use of different APIs on the same server by running multiple containers with different configurations. For example, one might run an OpenVINO container in addition to a CUDA container, or run a standard release container to maximize both CPU and GPU utilization.
+
+:::tip
+The machine learning container can be shared among several Immich instances regardless of the models a particular instance uses. However, using different models will lead to higher peak memory usage.
 :::

docs/docs/install/config-file.md

@@ -83,7 +83,7 @@ The default configuration looks like this:
   },
   "machineLearning": {
     "enabled": true,
-    "url": "http://immich-machine-learning:3003",
+    "url": ["http://immich-machine-learning:3003"],
     "clip": {
       "enabled": true,
       "modelName": "ViT-B-32__openai"

i18n/en.json

@@ -25,6 +25,7 @@
   "add_to": "Add to...",
   "add_to_album": "Add to album",
   "add_to_shared_album": "Add to shared album",
+  "add_url": "Add URL",
   "added_to_archive": "Added to archive",
   "added_to_favorites": "Added to favorites",
   "added_to_favorites_count": "Added {count, number} to favorites",
@@ -132,7 +133,7 @@
     "machine_learning_smart_search_description": "Search for images semantically using CLIP embeddings",
     "machine_learning_smart_search_enabled": "Enable smart search",
     "machine_learning_smart_search_enabled_description": "If disabled, images will not be encoded for smart search.",
-    "machine_learning_url_description": "URL of the machine learning server",
+    "machine_learning_url_description": "The URL of the machine learning server. If more than one URL is provided, each server will be attempted one-at-a-time until one responds successfully, in order from first to last.",
     "manage_concurrency": "Manage Concurrency",
     "manage_log_settings": "Manage log settings",
     "map_dark_style": "Dark style",
@@ -1045,6 +1046,7 @@
   "remove_from_album": "Remove from album",
   "remove_from_favorites": "Remove from favorites",
   "remove_from_shared_link": "Remove from shared link",
+  "remove_url": "Remove URL",
   "remove_user": "Remove user",
   "removed_api_key": "Removed API Key: {name}",
   "removed_from_archive": "Removed from archive",

open-api/immich-openapi-specs.json

@@ -11857,7 +11857,17 @@
             "$ref": "#/components/schemas/FacialRecognitionConfig"
           },
           "url": {
+            "deprecated": true,
+            "description": "This property was deprecated in v1.122.0",
             "type": "string"
+          },
+          "urls": {
+            "items": {
+              "format": "uri",
+              "type": "string"
+            },
+            "minItems": 1,
+            "type": "array"
           }
         },
         "required": [
@@ -11865,7 +11875,7 @@
           "duplicateDetection",
           "enabled",
           "facialRecognition",
-          "url"
+          "urls"
         ],
         "type": "object"
       },

open-api/typescript-sdk/src/fetch-client.ts

@@ -1178,7 +1178,9 @@ export type SystemConfigMachineLearningDto = {
     duplicateDetection: DuplicateDetectionConfig;
     enabled: boolean;
     facialRecognition: FacialRecognitionConfig;
-    url: string;
+    /** This property was deprecated in v1.122.0 */
+    url?: string;
+    urls: string[];
 };
 export type SystemConfigMapDto = {
     darkStyle: string;

server/src/config.ts

@@ -52,7 +52,7 @@ export interface SystemConfig {
   };
   machineLearning: {
     enabled: boolean;
-    url: string;
+    urls: string[];
     clip: {
       enabled: boolean;
       modelName: string;
@@ -206,7 +206,7 @@ export const defaults = Object.freeze<SystemConfig>({
   },
   machineLearning: {
     enabled: process.env.IMMICH_MACHINE_LEARNING_ENABLED !== 'false',
-    url: process.env.IMMICH_MACHINE_LEARNING_URL || 'http://immich-machine-learning:3003',
+    urls: [process.env.IMMICH_MACHINE_LEARNING_URL || 'http://immich-machine-learning:3003'],
     clip: {
       enabled: true,
       modelName: 'ViT-B-32__openai',

server/src/cores/storage.core.spec.ts

@@ -3,6 +3,8 @@ import { vitest } from 'vitest';
 
 vitest.mock('src/constants', () => ({
   APP_MEDIA_LOCATION: '/photos',
+  ADDED_IN_PREFIX: 'This property was added in ',
+  DEPRECATED_IN_PREFIX: 'This property was deprecated in ',
 }));
 
 describe('StorageCore', () => {

server/src/dtos/system-config.dto.ts

@@ -1,6 +1,7 @@
 import { ApiProperty } from '@nestjs/swagger';
-import { Type } from 'class-transformer';
+import { Exclude, Transform, Type } from 'class-transformer';
 import {
+  ArrayMinSize,
   IsBoolean,
   IsEnum,
   IsInt,
@@ -16,6 +17,7 @@ import {
   ValidateNested,
 } from 'class-validator';
 import { SystemConfig } from 'src/config';
+import { PropertyLifecycle } from 'src/decorators';
 import { CLIPConfig, DuplicateDetectionConfig, FacialRecognitionConfig } from 'src/dtos/model-config.dto';
 import {
   AudioCodec,
@@ -269,9 +271,16 @@ class SystemConfigMachineLearningDto {
   @ValidateBoolean()
   enabled!: boolean;
 
-  @IsUrl({ require_tld: false, allow_underscores: true })
+  @PropertyLifecycle({ deprecatedAt: 'v1.122.0' })
+  @Exclude()
+  url?: string;
+
+  @IsUrl({ require_tld: false, allow_underscores: true }, { each: true })
+  @ArrayMinSize(1)
+  @Transform(({ obj, value }) => (obj.url ? [obj.url] : value))
   @ValidateIf((dto) => dto.enabled)
-  url!: string;
+  @ApiProperty({ type: 'array', items: { type: 'string', format: 'uri' }, minItems: 1 })
+  urls!: string[];
 
   @Type(() => CLIPConfig)
   @ValidateNested()

server/src/interfaces/machine-learning.interface.ts

@@ -51,7 +51,7 @@ export type DetectedFaces = { faces: Face[] } & VisualResponse;
 export type MachineLearningRequest = ClipVisualRequest | ClipTextualRequest | FacialRecognitionRequest;
 
 export interface IMachineLearningRepository {
-  encodeImage(url: string, imagePath: string, config: ModelOptions): Promise<number[]>;
-  encodeText(url: string, text: string, config: ModelOptions): Promise<number[]>;
-  detectFaces(url: string, imagePath: string, config: FaceDetectionOptions): Promise<DetectedFaces>;
+  encodeImage(urls: string[], imagePath: string, config: ModelOptions): Promise<number[]>;
+  encodeText(urls: string[], text: string, config: ModelOptions): Promise<number[]>;
+  detectFaces(urls: string[], imagePath: string, config: FaceDetectionOptions): Promise<DetectedFaces>;
 }

server/src/migrations/1733339482860-RenameMachineLearningUrlToUrls.ts

@@ -0,0 +1,19 @@
+import { MigrationInterface, QueryRunner } from 'typeorm';
+
+export class RenameMachineLearningUrlToUrls1733339482860 implements MigrationInterface {
+  public async up(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(`
+        UPDATE system_metadata
+        SET value = jsonb_insert(value #- '{machineLearning,url}', '{machineLearning,urls}'::text[], jsonb_build_array(value->'machineLearning'->'url'))
+        WHERE key = 'system-config' AND value->'machineLearning'->'url' IS NOT NULL
+    `);
+  }
+
+  public async down(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(`
+        UPDATE system_metadata
+        SET value = jsonb_insert(value #- '{machineLearning,urls}', '{machineLearning,url}'::text[], to_jsonb(value->'machineLearning'->'urls'->>0))
+        WHERE key = 'system-config' AND value->'machineLearning'->'urls' IS NOT NULL AND jsonb_array_length(value->'machineLearning'->'urls') >= 1
+    `);
+  }
+}

server/src/repositories/event.repository.ts

@@ -155,7 +155,7 @@ export class EventRepository implements OnGatewayConnection, OnGatewayDisconnect
     this.emitHandlers[event].push(item);
   }
 
-  async emit<T extends EmitEvent>(event: T, ...args: ArgsOf<T>): Promise<void> {
+  emit<T extends EmitEvent>(event: T, ...args: ArgsOf<T>): Promise<void> {
     return this.onEvent({ name: event, args, server: false });
   }
 

server/src/repositories/machine-learning.repository.ts

@@ -1,6 +1,7 @@
-import { Injectable } from '@nestjs/common';
+import { Inject, Injectable } from '@nestjs/common';
 import { readFile } from 'node:fs/promises';
 import { CLIPConfig } from 'src/dtos/model-config.dto';
+import { ILoggerRepository } from 'src/interfaces/logger.interface';
 import {
   ClipTextualResponse,
   ClipVisualResponse,
@@ -13,33 +14,42 @@ import {
   ModelType,
 } from 'src/interfaces/machine-learning.interface';
 
-const errorPrefix = 'Machine learning request';
-
 @Injectable()
 export class MachineLearningRepository implements IMachineLearningRepository {
-  private async predict<T>(url: string, payload: ModelPayload, config: MachineLearningRequest): Promise<T> {
+  constructor(@Inject(ILoggerRepository) private logger: ILoggerRepository) {
+    this.logger.setContext(MachineLearningRepository.name);
+  }
+
+  private async predict<T>(urls: string[], payload: ModelPayload, config: MachineLearningRequest): Promise<T> {
     const formData = await this.getFormData(payload, config);
+    for (const url of urls) {
+      try {
+        const response = await fetch(new URL('/predict', url), { method: 'POST', body: formData });
+        if (response.ok) {
+          return response.json();
+        }
 
-    const res = await fetch(new URL('/predict', url), { method: 'POST', body: formData }).catch(
-      (error: Error | any) => {
-        throw new Error(`${errorPrefix} to "${url}" failed with ${error?.cause || error}`);
-      },
+        this.logger.warn(
+          `Machine learning request to "${url}" failed with status ${response.status}: ${response.statusText}`,
+        );
+      } catch (error: Error | unknown) {
+        this.logger.warn(
+          `Machine learning request to "${url}" failed: ${error instanceof Error ? error.message : error}`,
         );
-
-    if (res.status >= 400) {
-      throw new Error(`${errorPrefix} '${JSON.stringify(config)}' failed with status ${res.status}: ${res.statusText}`);
       }
-    return res.json();
     }
 
-  async detectFaces(url: string, imagePath: string, { modelName, minScore }: FaceDetectionOptions) {
+    throw new Error(`Machine learning request '${JSON.stringify(config)}' failed for all URLs`);
+  }
+
+  async detectFaces(urls: string[], imagePath: string, { modelName, minScore }: FaceDetectionOptions) {
     const request = {
       [ModelTask.FACIAL_RECOGNITION]: {
         [ModelType.DETECTION]: { modelName, options: { minScore } },
         [ModelType.RECOGNITION]: { modelName },
       },
     };
-    const response = await this.predict<FacialRecognitionResponse>(url, { imagePath }, request);
+    const response = await this.predict<FacialRecognitionResponse>(urls, { imagePath }, request);
     return {
       imageHeight: response.imageHeight,
       imageWidth: response.imageWidth,
@@ -47,15 +57,15 @@ export class MachineLearningRepository implements IMachineLearningRepository {
     };
   }
 
-  async encodeImage(url: string, imagePath: string, { modelName }: CLIPConfig) {
+  async encodeImage(urls: string[], imagePath: string, { modelName }: CLIPConfig) {
     const request = { [ModelTask.SEARCH]: { [ModelType.VISUAL]: { modelName } } };
-    const response = await this.predict<ClipVisualResponse>(url, { imagePath }, request);
+    const response = await this.predict<ClipVisualResponse>(urls, { imagePath }, request);
     return response[ModelTask.SEARCH];
   }
 
-  async encodeText(url: string, text: string, { modelName }: CLIPConfig) {
+  async encodeText(urls: string[], text: string, { modelName }: CLIPConfig) {
     const request = { [ModelTask.SEARCH]: { [ModelType.TEXTUAL]: { modelName } } };
-    const response = await this.predict<ClipTextualResponse>(url, { text }, request);
+    const response = await this.predict<ClipTextualResponse>(urls, { text }, request);
     return response[ModelTask.SEARCH];
   }
 

server/src/services/person.service.spec.ts

@@ -717,7 +717,7 @@ describe(PersonService.name, () => {
       assetMock.getByIds.mockResolvedValue([assetStub.image]);
       await sut.handleDetectFaces({ id: assetStub.image.id });
       expect(machineLearningMock.detectFaces).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ minScore: 0.7, modelName: 'buffalo_l' }),
       );

server/src/services/person.service.ts

@@ -297,7 +297,7 @@ export class PersonService extends BaseService {
     }
 
     const { imageHeight, imageWidth, faces } = await this.machineLearningRepository.detectFaces(
-      machineLearning.url,
+      machineLearning.urls,
       previewFile.path,
       machineLearning.facialRecognition,
     );

server/src/services/search.service.ts

@@ -86,7 +86,7 @@ export class SearchService extends BaseService {
     const userIds = await this.getUserIdsToSearch(auth);
 
     const embedding = await this.machineLearningRepository.encodeText(
-      machineLearning.url,
+      machineLearning.urls,
       dto.query,
       machineLearning.clip,
     );

server/src/services/smart-info.service.spec.ts

@@ -289,7 +289,7 @@ describe(SmartInfoService.name, () => {
       expect(await sut.handleEncodeClip({ id: assetStub.image.id })).toEqual(JobStatus.SUCCESS);
 
       expect(machineLearningMock.encodeImage).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ modelName: 'ViT-B-32__openai' }),
       );
@@ -322,7 +322,7 @@ describe(SmartInfoService.name, () => {
 
       expect(databaseMock.wait).toHaveBeenCalledWith(512);
       expect(machineLearningMock.encodeImage).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ modelName: 'ViT-B-32__openai' }),
       );

server/src/services/smart-info.service.ts

@@ -122,7 +122,7 @@ export class SmartInfoService extends BaseService {
     }
 
     const embedding = await this.machineLearningRepository.encodeImage(
-      machineLearning.url,
+      machineLearning.urls,
       previewFile.path,
       machineLearning.clip,
     );

server/src/services/system-config.service.spec.ts

@@ -85,7 +85,7 @@ const updatedConfig = Object.freeze<SystemConfig>({
   },
   machineLearning: {
     enabled: true,
-    url: 'http://immich-machine-learning:3003',
+    urls: ['http://immich-machine-learning:3003'],
     clip: {
       enabled: true,
       modelName: 'ViT-B-32__openai',
@@ -330,11 +330,11 @@ describe(SystemConfigService.name, () => {
 
     it('should allow underscores in the machine learning url', async () => {
       configMock.getEnv.mockReturnValue(mockEnvData({ configFile: 'immich-config.json' }));
-      const partialConfig = { machineLearning: { url: 'immich_machine_learning' } };
+      const partialConfig = { machineLearning: { urls: ['immich_machine_learning'] } };
       systemMock.readFile.mockResolvedValue(JSON.stringify(partialConfig));
 
       const config = await sut.getSystemConfig();
-      expect(config.machineLearning.url).toEqual('immich_machine_learning');
+      expect(config.machineLearning.urls).toEqual(['immich_machine_learning']);
     });
 
     const externalDomainTests = [

web/src/lib/components/admin-page/settings/machine-learning-settings/machine-learning-settings.svelte

@@ -12,6 +12,9 @@
   import { t } from 'svelte-i18n';
   import FormatMessage from '$lib/components/i18n/format-message.svelte';
   import { SettingInputFieldType } from '$lib/constants';
+  import Button from '$lib/components/elements/buttons/button.svelte';
+  import CircleIconButton from '$lib/components/elements/buttons/circle-icon-button.svelte';
+  import { mdiMinusCircle } from '@mdi/js';
 
   interface Props {
     savedConfig: SystemConfigDto;
@@ -42,15 +45,42 @@
 
         <hr />
 
+        <div>
+          {#each config.machineLearning.urls as _, i}
+            {#snippet removeButton()}
+              {#if config.machineLearning.urls.length > 1}
+                <CircleIconButton
+                  size="24"
+                  class="ml-2"
+                  padding="2"
+                  color="red"
+                  title=""
+                  onclick={() => config.machineLearning.urls.splice(i, 1)}
+                  icon={mdiMinusCircle}
+                />
+              {/if}
+            {/snippet}
+
             <SettingInputField
               inputType={SettingInputFieldType.TEXT}
-          label={$t('url')}
-          description={$t('admin.machine_learning_url_description')}
-          bind:value={config.machineLearning.url}
-          required={true}
+              label={i === 0 ? $t('url') : undefined}
+              description={i === 0 ? $t('admin.machine_learning_url_description') : undefined}
+              bind:value={config.machineLearning.urls[i]}
+              required={i === 0}
               disabled={disabled || !config.machineLearning.enabled}
-          isEdited={config.machineLearning.url !== savedConfig.machineLearning.url}
+              isEdited={i === 0 && !isEqual(config.machineLearning.urls, savedConfig.machineLearning.urls)}
+              trailingSnippet={removeButton}
             />
+          {/each}
+        </div>
+
+        <Button
+          class="mb-2"
+          type="button"
+          size="sm"
+          onclick={() => config.machineLearning.urls.splice(0, 0, '')}
+          disabled={disabled || !config.machineLearning.enabled}>{$t('add_url')}</Button
+        >
       </div>
 
       <SettingAccordion

web/src/lib/components/elements/buttons/circle-icon-button.svelte

@@ -1,5 +1,5 @@
 <script lang="ts" module>
-  export type Color = 'transparent' | 'light' | 'dark' | 'gray' | 'primary' | 'opaque' | 'alert';
+  export type Color = 'transparent' | 'light' | 'dark' | 'red' | 'gray' | 'primary' | 'opaque' | 'alert';
   export type Padding = '1' | '2' | '3';
 </script>
 
@@ -64,6 +64,7 @@
     transparent: 'bg-transparent hover:bg-[#d3d3d3] dark:text-immich-dark-fg',
     opaque: 'bg-transparent hover:bg-immich-bg/30 text-white hover:dark:text-white',
     light: 'bg-white hover:bg-[#d3d3d3]',
+    red: 'text-red-400 hover:bg-[#d3d3d3]',
     dark: 'bg-[#202123] hover:bg-[#d3d3d3]',
     alert: 'text-[#ff0000] hover:text-white',
     gray: 'bg-[#d3d3d3] hover:bg-[#e2e7e9] text-immich-dark-gray hover:text-black',

web/src/lib/components/shared-components/settings/setting-input-field.svelte

@@ -22,6 +22,7 @@
     autofocus?: boolean;
     passwordAutocomplete?: AutoFill;
     descriptionSnippet?: Snippet;
+    trailingSnippet?: Snippet;
   }
 
   let {
@@ -39,6 +40,7 @@
     autofocus = false,
     passwordAutocomplete = 'current-password',
     descriptionSnippet,
+    trailingSnippet,
   }: Props = $props();
 
   let input: HTMLInputElement | undefined = $state();
@@ -68,7 +70,7 @@
 </script>
 
 <div class="mb-4 w-full">
-  <div class={`flex h-[26px] place-items-center gap-1`}>
+  <div class={`flex place-items-center gap-1`}>
     <label class="font-medium text-immich-primary dark:text-immich-dark-primary text-sm" for={label}>{label}</label>
     {#if required}
       <div class="text-red-400">*</div>
@@ -132,6 +134,8 @@
         {disabled}
         {title}
       />
+
+      {@render trailingSnippet?.()}
     </div>
   {:else}
     <PasswordField