1
0
mirror of https://github.com/immich-app/immich.git synced 2024-11-24 08:52:28 +02:00
Commit Graph

287 Commits

Author SHA1 Message Date
Alexandre Bouijoux
b8777d7739
Add french documentation (#4010)
* Copy original README for online editing

* Translate to french

* Add link from other documentations

* Add machine-learning french doc

* Typos

* Missing word

---------

Co-authored-by: Alexandre Bouijoux <alexandre@bouijoux.fr>
2023-09-08 13:48:39 +07:00
Mert
0a24ff90bb
fix(ml): set higher default worker timeout (#4007) 2023-09-07 08:27:29 +07:00
Alex The Bot
608543da0b Version v1.77.0 2023-09-06 03:30:44 +00:00
Mert
3cf0f5f11b
fix(ml): model downloading improvements (#3988)
* handle ort `NoSuchFile` error, stricter file check

* keep exception order
2023-09-06 08:48:40 +07:00
Mert
b7fd5dcb4a
dev(ml): fixed docker-compose.dev.yml, updated locust (#3951)
* fixed dev docker compose

* updated locustfile

* deleted old script, moved comments to locustfile
2023-09-01 21:59:17 -04:00
Daniele Ricci
c7d53a5006
docs: remove obsolete environment variables (#3926)
* Obsolete environment variables

* Review fixes

* Review fixes

* Review fixes
2023-09-01 18:05:45 +07:00
Mert
41461e0d5d
chore(ml): memory optimisations (#3934) 2023-08-31 18:30:53 -05:00
Alex The Bot
58ae734fc2 Version v1.76.1 2023-08-30 08:26:04 +00:00
Mert
54b2779b79
chore(ml): improved logging (#3918)
* fixed `minScore` not being set correctly

* apply to init

* don't send `enabled`

* fix eslint warning

* added logger

* added logging

* refinements

* enable access log for info level

* formatting

* merged strings

---------

Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-08-30 08:22:01 +00:00
Mert
df26e12db6
fix(ml): minScore not being set correctly (#3916)
* fixed `minScore` not being set correctly

* apply to init

* don't send `enabled`

* fix eslint warning

* better error message
2023-08-30 03:16:00 -05:00
Alex The Bot
49c2d4d115 Version v1.76.0 2023-08-29 19:24:43 +00:00
Mert
bcc36d14a1
feat(ml)!: customizable ML settings (#3891)
* consolidated endpoints, added live configuration

* added ml settings to server

* added settings dashboard

* updated deps, fixed typos

* simplified modelconfig

updated tests

* Added ml setting accordion for admin page

updated tests

* merge `clipText` and `clipVision`

* added face distance setting

clarified setting

* add clip mode in request, dropdown for face models

* polished ml settings

updated descriptions

* update clip field on error

* removed unused import

* add description for image classification threshold

* pin safetensors for arm wheel

updated poetry lock

* moved dto

* set model type only in ml repository

* revert form-data package install

use fetch instead of axios

* added slotted description with link

updated facial recognition description

clarified effect of disabling tasks

* validation before model load

* removed unnecessary getconfig call

* added migration

* updated api

updated api

updated api

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-08-29 08:58:00 -05:00
Alex The Bot
f1027d7807 Version v1.75.2 2023-08-26 22:34:54 +00:00
Alex The Bot
cc1fecfffd Version v1.75.1 2023-08-26 18:31:14 +00:00
Alex The Bot
6fe214a784 Version v1.75.0 2023-08-26 04:44:39 +00:00
Mert
165b91b068
feat(ml)!: switch image classification and CLIP models to ONNX (#3809) 2023-08-25 06:28:51 +02:00
Mert
60729a091a
make lazy loading default (#3797) 2023-08-20 18:24:14 -05:00
Alex The Bot
d2bad1d553 Version v1.74.0 2023-08-19 06:09:16 +00:00
Alex The Bot
a3b6095b61 Version v1.73.0 2023-08-15 12:45:48 +00:00
Mert
fa57853bd2
fix(ml): clear cache if bad zip file (#3600) 2023-08-08 12:33:13 -05:00
Alex The Bot
6efc2ec9be Version v1.72.2 2023-08-07 16:41:17 +00:00
Alex The Bot
b3b5f063cf Version v1.72.1 2023-08-07 02:54:24 +00:00
Alex The Bot
3edb347666 Version v1.72.0 2023-08-06 19:48:25 +00:00
Mert
c73832bd9c
refactor(ml): model downloading (#3545)
* download facial recognition models

* download hf models

* simplified logic

* updated `predict` for facial recognition

* ensure download method is called

* fixed repo_id for clip

* fixed download destination

* use st's own `snapshot_download`

* conditional download

* fixed predict method

* check if loaded

* minor fixes

* updated mypy overrides

* added pytest-mock

* updated tests

* updated lock
2023-08-05 21:45:13 -05:00
Jorge Avila
1cf3378499
Add Spanish translations of Readme (#3511) 2023-08-02 06:51:08 -05:00
Alex The Bot
3f1cf44717 Version v1.71.0 2023-07-28 18:32:33 +00:00
Alex The Bot
2872886e77 Version v1.70.0 2023-07-27 03:40:16 +00:00
Alex The Bot
f41e1159d1 Version v1.69.0 2023-07-23 17:40:02 +00:00
Mert
8f08100a30
moved deps (#3379) 2023-07-22 15:04:52 -05:00
bt90
bc885f3644
fix(server): properly handle SIGTERM (#3350)
* use tini init

* Move python into CMD

* Use tini as entrypoint

* Toggle executable bit

* Avoid compose changes

* Adapt web entrypoint
2023-07-21 09:20:04 -05:00
Alex The Bot
1835fbae49 Version v1.68.0 2023-07-20 03:06:12 +00:00
Alex The Bot
81e07fda08 Version v1.67.2 2023-07-14 14:01:20 +00:00
Alex The Bot
2d4e2af629 Version v1.67.1 2023-07-14 02:45:02 +00:00
Alex The Bot
cd184cf366 Version v1.67.0 2023-07-14 00:39:54 +00:00
Mert
848ba685eb
fix(ml): race condition when loading models (#3207)
* sync model loading, disabled model ttl by default

* disable revalidation if model unloading disabled

* moved lock
2023-07-11 12:01:21 -05:00
Alex The Bot
0d30ceb284 Version v1.66.1 2023-07-05 02:50:53 +00:00
Alex The Bot
8a3ab5be3e Version v1.66.0 2023-07-04 15:51:53 +00:00
Alex The Bot
df9c05bef3 Version v1.65.0 2023-06-30 03:01:48 +00:00
Mert
df1e8679d9
chore(ml): added testing and github workflow (#2969)
* added testing

* github action for python, made mypy happy

* formatted with black

* minor fixes and styling

* test model cache

* cache test dependencies

* narrowed model cache tests

* moved endpoint tests to their own class

* cleaned up fixtures

* formatting

* removed unused dep
2023-06-27 18:21:33 -05:00
Mert
47982641b2
fix(ml): clear model cache on load error (#2951)
* clear model cache on load error

* updated caught exceptions
2023-06-27 16:01:24 -05:00
Mert
4d3ce0a65e
fixed setting different clip, removed unused stubs (#2987) 2023-06-27 12:21:50 -05:00
Alex The Bot
b258f3552a Version v1.64.0 2023-06-26 18:06:11 +00:00
Mert
a58482cb2b
added locustfile (#2926) 2023-06-25 13:20:45 -05:00
Mert
a2f5674bbb
refactor(ml): modularization and styling (#2835)
* basic refactor and styling

* removed batching

* module entrypoint

* removed unused imports

* model superclass,  model cache now in app state

* fixed cache dir and enforced abstract method

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-06-24 22:18:09 -05:00
Mert
837ad24f58
fix(deps): install poetry in pump workflow (#2938)
* install poetry in pump workflow

* formatting
2023-06-24 22:03:50 -05:00
Zeeshan Khan
34201be74c
feat(ml) backend takes image over HTTP (#2783)
* using pydantic BaseSetting

* ML API takes image file as input

* keeping image in memory

* reducing duplicate code

* using bytes instead of UploadFile & other small code improvements

* removed form-multipart, using HTTP body

* format code

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-06-17 22:49:19 -05:00
renovate[bot]
0909335d02
chore(deps): update python:3.11.4-slim-bullseye docker digest to 91d194f (#2797)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-06-16 11:38:59 -05:00
renovate[bot]
827e4b5f75
chore(deps): update python:3.11.4-bullseye docker digest to 5b40167 (#2796)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-06-16 11:17:23 -05:00
Thomas
16171eee8d
pin image digests (#2754)
Manifest list digests can be found with:

```sh
docker buildx imagetools inspect python:3.11.4-bullseye
docker buildx imagetools inspect python:3.11.4-slim-bullseye
docker buildx imagetools inspect ghcr.io/nginxinc/nginx-unprivileged:1.25.0-alpine3.17
```

The node images are pinned in #2736

Fixes #2751
Partially fixes #2752
2023-06-16 10:28:41 -05:00
Mert
d0cc231782
feat(ml): model unloading (#2661)
* model cache

* fixed revalidation when using cache namespace

* fixed ttl not being set, added lock
2023-06-06 20:48:51 -05:00
Mert
1e748864c5
chore(ml): updated dockerfile, added typing, packaging (#2642)
* updated dockerfile, added typing, packaging

apply env change

* added arm64 support

* added ml version pump, second try for arm64

* added linting config to pyproject.toml

* renamed ml input field

* fixed linter config

* fixed dev docker compose
2023-06-05 09:40:48 -05:00
Mert
b8de668f5f
feat(ml): env variables for tags, faces and eager startup (#2626)
* env variables for tags, faces and eager startup

* chore(server,ml): remove object detection job and endpoint (#2627)

* removed object detection job

* removed object detection endpoint

* env variables for tags, faces and eager startup

* download without caching models if not eager

* simplified `get_cached_model`

* re-added env for clip text model
2023-06-02 21:42:47 -05:00
Mert
631f13cf2f
chore(server,ml): remove object detection job and endpoint (#2627)
* removed object detection job

* removed object detection endpoint
2023-05-31 20:49:51 -05:00
Alex
84cfa38510
chore(ml): load models on start up (#2487)
* chore(ml): load models on start up

* Download correct model
2023-05-19 22:37:01 -05:00
Jason Rasmussen
93863b0629
feat: facial recognition (#2180) 2023-05-17 12:07:17 -05:00
Alex
8cc9b08c06
chore(ml): use official pytorch channel (#2416) 2023-05-10 07:09:33 -05:00
bo0tzz
b6b9f51bd7
chore(ml): Fix entrypoint path (#2373) 2023-05-03 14:27:35 -05:00
Alex
7e965cb6d4
chore(ml): move to fastAPI (#2336) 2023-04-26 05:39:24 -05:00
Alex Tran
08b424b3df update gitignore 2023-04-22 13:31:42 -05:00
Devin Buhl
2dcccb37a0
chore(docker): Default NODE_ENV to production for server image and update alpine version (#2157)
* default NODE_ENV to production for server image

* update node image to use 3.17 alpine in server

* update web docker image to use alpine 3.17

* remove NODE_ENV from production docker-compose

* NODE_ENV is also needed default in machine-learning
2023-04-03 15:05:29 -05:00
Alex
f56eaae019
feat(server): CLIP search integration (#1939) 2023-03-18 08:44:42 -05:00
Olly Welch
d5d0624311
Feat/ml image optimisations (#1916)
* Use multi stage build to slim down ML image size

* Use gunicorn as WSGI server in ML image

* Configure gunicorn server for ML use case

* Use requirements.txt file to install python dependencies in ML image

* Make ML listen IP configurable

* Revert "Use requirements.txt file to install python dependencies in ML image"

This reverts commit 32e706c7f3.

* Separate out pip installs in ML builder image
2023-03-03 16:45:20 -06:00
Zack Pollard
1cc184ed10
Revert "feat(server): Machine learning's image optimisations (#1908)" (#1915)
This reverts commit 977740045a.
2023-03-01 11:48:35 -06:00
Olly Welch
977740045a
feat(server): Machine learning's image optimisations (#1908)
* Use multi stage build to slim down ML image size

* Use gunicorn as WSGI server in ML image

* Configure gunicorn server for ML use case

* Use requirements.txt file to install python dependencies in ML image

* Make ML listen IP configurable
2023-03-01 09:37:12 -06:00
Olly Welch
ab90b01122
feat(server) Enable ML support for ARM CPUs (#1880)
* Install nightly release of pytorch to enable ML support for arm CPUs

* Remove linux/arm/v7 from ML docker builds

* Add --no-cache-dir to torch installation command in ML image build

* Use PIP_NO_CACHE_DIR option in ML build to further decrease image size
2023-02-26 15:00:09 -06:00
Zack Pollard
3d468c369c
fix: machine learning only take results with > 90% confidence (#1875) 2023-02-25 22:02:35 -06:00
Skyler Mäntysaari
b00631d186
fix(machine-learning): Add the command to execute at startup (#1843)
* fix(machine-learning): Add the command to execute at startup

Previously it wasn't set in the Docker container but it should be.

* fix(docker): remove machine-learning command arg

* fix(docker): machine-learning CMD argument
2023-02-23 09:54:04 -06:00
Alex
a3b9a0be3a
fix(server): Flask not found (#1830)
* fix(server): Flask not found

* trial 1
2023-02-22 10:53:59 -06:00
Alex
57136e48fb
feat(machine-learning)!: move machine learning to Python based image (#1774)
BREAKING CHANGES
* Users have to update the docker-compose file, machine-learning portion.
* Temporary dropping machine-learning support for Arm64 and Armv7
2023-02-18 09:13:37 -06:00
Zack Pollard
43da8c2a72
chore: reduce docker image size (#1523)
* chore: remove @tensorflow/tfjs-node-gpu as it is unused

* chore: remove ffmpeg from machine-learning docker image

* chore: remove unneeded dependencies + move dev dependencies in server

* chore: reduce server image size

* chore: machine-learning remove extraneous dependencies

* chore: web remove extraneous dependencies

* chore: web Dockerfile reduce production image size

* chore: add exiftool-vendored.pl as a dependency
2023-02-02 21:28:34 -06:00
Skyler Mäntysaari
9d337bf4dc
feat(server/machine-learning): Configurable port (#1386)
* feat(server/machine-learning): Configurable port

* feat(server/machine-learning): Address PR comments.

* feat(server/machine-learning): Simplify

Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-01-23 22:18:35 -06:00
bt90
16b763e086
fix(docker-build): start main process with exec (#1210)
* Use exec

* Appl shellcheck fixes

* Close with newline
2022-12-29 14:49:02 -06:00
Jason Rasmussen
2e4c005ad9
refactor: multistage builds (#955) 2022-11-10 22:22:17 -06:00
Alex Tran
c3348bd068
Fixed Dockerfile not working in dev 2022-11-09 23:34:35 -06:00
Jason Rasmussen
cc61729f01
build(server): use github-action cache (#949)
* build(server): prune dependencies in docker builder

* fix: e2e tests

* refactor: dockerfile step order

* fix: vips build dependency

* feat: use caching
2022-11-09 19:53:21 -06:00
Alex
676ad2d34f
chore(server) revert Dockerfile (#878) 2022-10-27 17:21:28 -05:00
Jonas Janz
b713fb5650
feat(docker) revert ubuntu base image (#863)
* feat(docker) revert ubuntu base image

This PR reverts the base image for immich-server back to alpine

Adds LICENSE to all Images
Quiets apt-get commands when building
ensures write-permission for root group on app folders

Signed-off-by: PixelJonas <5434875+PixelJonas@users.noreply.github.com>

* Test build old Docker content

* Revert and retry

Signed-off-by: PixelJonas <5434875+PixelJonas@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-10-25 14:18:37 -05:00
Alex
c3979f6e31
fix(machine-learning) Remove unsused database config (#745) 2022-09-23 19:00:47 -05:00
Alex
c436c57cc9
Fixed immich-machine-learning container not starting correctly in production 2022-08-20 23:04:10 -07:00
Alex
10ef3509dd
Fixed machine-learning container cannot start prod 2022-08-20 22:27:25 -07:00
Alex
3dc538f9e6
Fixed machine-learning container cannot start prod 2022-08-20 22:26:47 -07:00
Thanh Pham
1e29ff322d
build(server): minimal container (#506)
* build(server): update Dockerfile

* build(server): fix dockerfile

* build(machine-learning): multiple build stages

* build(server): update Dockerfile
2022-08-20 21:19:02 -07:00
bo0tzz
a35460cb84
Bump tfjs version to 3.19.0 for arm64 support (#368)
* Add linux/arm64 to machine-learning container build

* Bump tfjs version to 3.19.0

* Fix tfjs dependency error
2022-07-23 14:15:55 -05:00
Fynn Petersen-Frey
d69470e207
Add extended redis & DB port configuration via environment variables (#330)
* Add database port configuration via env variable

Add redis connection configuration via env variables

* Add redis connection configuration via env variables
2022-07-12 22:21:11 -05:00
Jaime Baez
355038a91a
Use npm ci for installing pacakages (#304) 2022-07-04 13:47:25 -05:00
Zack Pollard
c071e64a7e
infra: switch port to 3003 for machine learning container (#290)
* infra: switch port to 3003 for machine learning container

fixes #289

* Changed port of machine-learning-endpoint to match with new port

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-07-01 10:20:04 -05:00
Alex
a8220172f8
WIP refactor container and queuing system (#206)
* refactor microservices to machine-learning

* Update tGithub issue template with correct task syntax

* Added microservices container

* Communicate between service based on queue system

* added dependency

* Fixed problem with having to import BullQueue into the individual service

* Added todo

* refactor server into monorepo with microservices

* refactor database and entity to library

* added simple migration

* Move migrations and database config to library

* Migration works in library

* Cosmetic change in logging message

* added user dto

* Fixed issue with testing not able to find the shared library

* Clean up library mapping path

* Added webp generator to microservices

* Update Github Action build latest

* Fixed issue NPM cannot install due to conflict witl Bull Queue

* format project with prettier

* Modified docker-compose file

* Add GH Action for Staging build:

* Fixed GH action job name

* Modified GH Action to only build & push latest when pushing to main

* Added Test 2e2 Github Action

* Added Test 2e2 Github Action

* Implemented microservice to extract exif

* Added cronjob to scan and generate webp thumbnail  at midnight

* Refactor to ireduce hit time to database when running microservices

* Added error handling to asset services that handle read file from disk

* Added video transcoding queue to process one video at a time

* Fixed loading spinner on web while loading covering the info panel

* Add mechanism to show new release announcement to web and mobile app (#209)

* Added changelog page

* Fixed issues based on PR comments

* Fixed issue with video transcoding run on the server

* Change entry point content for backward combatibility when starting up server

* Added announcement box

* Added error handling to failed silently when the app version checking is not able to make the request to GITHUB

* Added new version announcement overlay

* Update message

* Added messages

* Added logic to check and show announcement

* Add method to handle saving new version

* Added button to dimiss the acknowledge message

* Up version for deployment to the app store
2022-06-11 16:12:06 -05:00