Add `AlbumRepository` method to retrieve an album's asset ids, with an
optional parameter to only filter by the provided asset ids. With this,
we can now check asset membership using a single query.
When adding or removing assets to an album, checking whether each asset
is already present in the album now requires a single query, instead of
one query per asset.
Related to #4539 performance improvements.
Before:
```
// Asset membership and permissions check (2 queries per asset)
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "albums" "AlbumEntity" LEFT JOIN "albums_assets_assets" "AlbumEntity_AlbumEntity__AlbumEntity_assets" ON "AlbumEntity_AlbumEntity__AlbumEntity_assets"."albumsId"="AlbumEntity"."id" LEFT JOIN "assets" "AlbumEntity__AlbumEntity_assets" ON "AlbumEntity__AlbumEntity_assets"."id"="AlbumEntity_AlbumEntity__AlbumEntity_assets"."assetsId" AND ("AlbumEntity__AlbumEntity_assets"."deletedAt" IS NULL) WHERE ( ("AlbumEntity"."id" = $1 AND "AlbumEntity__AlbumEntity_assets"."id" = $2) ) AND ( "AlbumEntity"."deletedAt" IS NULL )) LIMIT 1 -- PARAMETERS: ["3fdf0e58-a1c7-4efe-8288-06e4c3f38df9","b666ae6c-afa8-4d6f-a1ad-7091a0659320"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["b666ae6c-afa8-4d6f-a1ad-7091a0659320","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "albums" "AlbumEntity" LEFT JOIN "albums_assets_assets" "AlbumEntity_AlbumEntity__AlbumEntity_assets" ON "AlbumEntity_AlbumEntity__AlbumEntity_assets"."albumsId"="AlbumEntity"."id" LEFT JOIN "assets" "AlbumEntity__AlbumEntity_assets" ON "AlbumEntity__AlbumEntity_assets"."id"="AlbumEntity_AlbumEntity__AlbumEntity_assets"."assetsId" AND ("AlbumEntity__AlbumEntity_assets"."deletedAt" IS NULL) WHERE ( ("AlbumEntity"."id" = $1 AND "AlbumEntity__AlbumEntity_assets"."id" = $2) ) AND ( "AlbumEntity"."deletedAt" IS NULL )) LIMIT 1 -- PARAMETERS: ["3fdf0e58-a1c7-4efe-8288-06e4c3f38df9","c656ab1c-7775-4ff7-b56f-01308c072a76"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["c656ab1c-7775-4ff7-b56f-01308c072a76","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "albums" "AlbumEntity" LEFT JOIN "albums_assets_assets" "AlbumEntity_AlbumEntity__AlbumEntity_assets" ON "AlbumEntity_AlbumEntity__AlbumEntity_assets"."albumsId"="AlbumEntity"."id" LEFT JOIN "assets" "AlbumEntity__AlbumEntity_assets" ON "AlbumEntity__AlbumEntity_assets"."id"="AlbumEntity_AlbumEntity__AlbumEntity_assets"."assetsId" AND ("AlbumEntity__AlbumEntity_assets"."deletedAt" IS NULL) WHERE ( ("AlbumEntity"."id" = $1 AND "AlbumEntity__AlbumEntity_assets"."id" = $2) ) AND ( "AlbumEntity"."deletedAt" IS NULL )) LIMIT 1 -- PARAMETERS: ["3fdf0e58-a1c7-4efe-8288-06e4c3f38df9","cf82adb2-1fcc-4f9e-9013-8fc03cc8d3a9"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["cf82adb2-1fcc-4f9e-9013-8fc03cc8d3a9","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
```
After:
```
// Asset membership check (1 query for all assets)
immich_server | query: SELECT "albums_assets"."assetsId" AS "assetId" FROM "albums_assets_assets" "albums_assets" WHERE "albums_assets"."albumsId" = $1 AND "albums_assets"."assetsId" IN ($2, $3, $4) -- PARAMETERS: ["ca870d76-6311-4e89-bf9a-f5b51ea2452c","b666ae6c-afa8-4d6f-a1ad-7091a0659320","c656ab1c-7775-4ff7-b56f-01308c072a76","cf82adb2-1fcc-4f9e-9013-8fc03cc8d3a9"]
// Permissions check (1 query per asset)
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["b666ae6c-afa8-4d6f-a1ad-7091a0659320","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["c656ab1c-7775-4ff7-b56f-01308c072a76","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
immich_server | query: SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (SELECT 1 FROM "assets" "AssetEntity" WHERE ("AssetEntity"."id" = $1 AND "AssetEntity"."ownerId" = $2)) LIMIT 1 -- PARAMETERS: ["cf82adb2-1fcc-4f9e-9013-8fc03cc8d3a9","6bc60cf1-bd18-4501-a1c2-120b51276fda"]
```
* fix: don't reveal user count publicly
* fix: mobile and user controller
* fix: update other frontend endpoints
* fix: revert openapi change
* chore: open api
* fix: initialize
* openapi
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* fix: inline mark asset as offline
* fix: improve log message
* chore: lint
* fix: offline asset algorithm
* fix: use set comparison to check what to import
* fix: only mark new offline files as offline
* fix: compare the correct array
* fix: set default library concurrency to 5
* fix: remove one db call when scanning new files
* chore: remove unused import
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* feat: improve performances in people page
* feat: add loadingspinner when searching
* fix: reset people on error
* fix: case insensitive
* feat: better sql query
* fix: reset people list before api request
* fix: format
* fix: timezone bucket timezones
* chore: open api
* fix: interpret local time in utc
* fix: tests
* fix: refactor memory lane
* fix(web): use local date in memory viewer
* chore: set localDateTime non-null
* fix: filter out memories from the current year
* wip: move localDateTime to asset
* fix: correct sorting from db
* fix: migration
* fix: web typo
* fix: formatting
* fix: e2e
* chore: localDateTime is non-null
* chore: more non-nulliness
* fix: asset stub
* fix: tests
* fix: use extract and index for day of year
* fix: don't show memories before today
* fix: cleanup
* fix: tests
* fix: only use localtime for tz
* fix: display memories in client timezone
* fix: tests
* fix: svelte tests
* fix: bugs
* chore: open api
---------
Co-authored-by: Jonathan Jogenfors <jonathan@jogenfors.se>
* Add include archive setting to map on web
* open api
* better naming for web isArchived variable
* add withArchived setting to mobile
* (e2e): tests for mapMarker endpoint and isArchived
* isArchived to mobile
* chore: cleanup test
* chore: optimize e2e
---------
Co-authored-by: shalong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* feat(server): get random assets API
* Fix tests
* Use correct validation annotation
* Fix offset use in query
* Update API specs
* Fix typo
* Random assets e2e tests
* Improve e2e tests
* use access core for all person methods
* minor fixes, feedback
* reorder assignments
* remove unnecessary permission requirement
* unify naming of tests
* reorder variables
* soft delete albums when user gets soft deleted
* fix wrong intl openapi version
* fix tests
* ability to restore albums, automatically restore when user restored
* (e2e) tests for shared albums via link and with user
* (e2e) test deletion of users and linked albums
* (e2e) fix share album with owner test
* fix: deletedAt
* chore: fix restore order
* fix: use timezone date column
* chore: cleanup e2e tests
* (e2e) fix user delete test
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* tests for person service
* tests for auth service
* tests for access core
* improve tests for album service
* fix missing brackets and remove comments
* tests for asset service
* tests for face recognition
* tests for job service
* feedback
* tests for search service (broken)
* fix: disabled search test
* tests for smart-info service
* tests for storage template service
* tests for user service
* fix formatting of untouched files LOL
* attempt to fix formatting
* streamline api utils, add asset api for uploading files
* test upload of assets
* fix formatting
* move test-utils to correct folder
* test add assets to album
* use random bytes instead of test image
* (e2e) test albums with assets
* (e2e) complete tests for album endpoints
* (e2e) tests for asset endpoint
* fix: asset upload/import dto validation
* (e2e) tests for statistics asset endpoint
* fix wrong describe text
* (e2e) tests for people with faces
* (e2e) clean up person tests
* (e2e) tests for partner sharing endpoints
* (e2e) tests for link sharing
* (e2e) tests for the asset time bucket endpoint
* fix minor issues
* remove access.core.spec.ts
* chore: wording
* chore: organize test api files
* chore: fix test describe
* implement feedback
* fix race condition in album tests
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* feat(web): show original uploader in shared album photo details
* feat: send owner in asset by id response
* chore: open api
* fix: linting
* fix: change to Shared By
* openapi
* openapi
* api
* styling
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex <alex.tran1502@gmail.com>
* custom `IsOptional`
* added link to source
* formatting
* Update server/src/domain/domain.util.ts
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* nullable birth date endpoint
* made `nullable` a property
* formatting
* removed unused dto
* updated decorator arg
* fixed album e2e tests
* add null tests for auth e2e
* add null test for person e2e
* fixed tests
* added null test for user e2e
* removed unusued import
* log key in test name
* chore: add note about mobile not being able to use the endpoint
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* consolidated endpoints, added live configuration
* added ml settings to server
* added settings dashboard
* updated deps, fixed typos
* simplified modelconfig
updated tests
* Added ml setting accordion for admin page
updated tests
* merge `clipText` and `clipVision`
* added face distance setting
clarified setting
* add clip mode in request, dropdown for face models
* polished ml settings
updated descriptions
* update clip field on error
* removed unused import
* add description for image classification threshold
* pin safetensors for arm wheel
updated poetry lock
* moved dto
* set model type only in ml repository
* revert form-data package install
use fetch instead of axios
* added slotted description with link
updated facial recognition description
clarified effect of disabling tasks
* validation before model load
* removed unnecessary getconfig call
* added migration
* updated api
updated api
updated api
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* implement method to read config file
* getConfig returns config file if present
* return isConfigFile for http requests
* disable elements if config file is used, show message if config file is set, copy existing config to clipboard
* fix allowing partial configuration files
* add new env variable to docs
* fix tests
* minor refactoring, address review
* adapt config type in frontend
* remove unnecessary imports
* move config file reading to system-config repo
* add documentation
* fix code formatting in system settings page
* add validator for config file
* fix formatting in docs
* update generated files
* throw error when trying to update config. e.g. via cli or api
* switch to feature flags for isConfigFile
* refactoring
* refactor: config file
* chore: open api
* feat: always show copy/export buttons
* fix: default flags
* refactor: copy to clipboard
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* Person birth date (data layer)
* Person birth date (data layer)
* Person birth date (service layer)
* Person birth date (service layer, API)
* Person birth date (service layer, API)
* Person birth date (UI) (wip)
* Person birth date (UI) (wip)
* Person birth date (UI) (wip)
* Person birth date (UI) (wip)
* UI: Use "date of birth" everywhere
* UI: better modal dialog
Similar to the API key modal.
* UI: set date of birth from people page
* Use typed events for modal dispatcher
* Date of birth tests (wip)
* Regenerate API
* Code formatting
* Fix Svelte typing
* Fix Svelte typing
* Fix person model [skip ci]
* Minor refactoring [skip ci]
* Typed event dispatcher [skip ci]
* Refactor typed event dispatcher [skip ci]
* Fix unchanged birthdate check [skip ci]
* Remove unnecessary custom transformer [skip ci]
* PersonUpdate: call search index update job only when needed
* Regenerate API
* Code formatting
* Fix tests
* Fix DTO
* Regenerate API
* chore: verbiage and view mode
* feat: show current age
* test: person e2e
* fix: show name for birth date selection
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* feat: server changes for album timeline
* feat(web): album timeline view
* chore: open api
* chore: remove archive action
* fix: favorite for non-owners
* feat(web,server): user preference for time-based memories
* chore: open api
* dev: mobile
* fix: update
* mobile work
---------
Co-authored-by: Alex <alex.tran1502@gmail.com>
Co-authored-by: Alex Tran <Alex.Tran@conductix.com>
* feat(server): add album description
* chore: open api
* fix: tests
* show and edit description on the web
* fix test
* remove unused code
* type event
* format fix
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* refactor: time buckets
* feat(web): use new time bucket api
* feat(web): use asset grid in archive/favorites
* chore: open api
* chore: clean up uuid validation
* refactor(web): move memory lane to photos page
* Update web/src/routes/(user)/archive/+page.svelte
Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>
* fix: hide archived photos on main timeline
* fix: select exif info
---------
Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>
* added transcode configs for nvenc,qsv and vaapi
* updated dev docker compose
* added software fallback
* working vaapi
* minor fixes and added tests
* updated api
* compile libvips
* move hwaccel settings to `hwaccel.yml`
* changed default dockerfile, moved `readdir` call
* removed unused import
* minor cleanup
* fix for arm build
* added documentation, minor fixes
* added intel driver
* updated docs
styling
* uppercase codec and api names
* formatting
* added tests
* updated docs
* removed semicolons
* added link to `hwaccel.yml`
* added newlines
* added `hwaccel` section to docker-compose.prod.yml
* ensure mesa drivers are installed
* switch to mimalloc for sharp
* moved build version and sha256 to json
* let libmfx set the render device
* possible fix for vp9 on qsv
* updated tests
* formatting
* review suggestions
* semicolon
* moved `LD_PRELOAD` to start script
* switched to jellyfin's ffmpeg package
* fixed dockerfile
* use cqp instead of icq for qsv vp9
* updated dockerfile
* added sha256sum for other platforms
* fixtures
* refactor: add/remove album assets
* chore: open api
* feat: remove owned assets from album
* refactor: move to bulk id req/res dto
* chore: open api
* chore: merge main
* dev: mobile work
* fix: adding asset from web not sync with mobile
* remove print statement
---------
Co-authored-by: Alex Tran <Alex.Tran@conductix.com>
* refactored `getFfmpegOptions`
refactor transcoding, make separate service
* fixed enum casing
* use `Logger` instead of `console.log`
* review suggestions
* use enum for `getHandler`
* fixed formatting
* Update server/src/domain/media/media.util.ts
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* Update server/src/domain/media/media.util.ts
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* More specific imports, renamed codec classes
* simplified code
* removed unused import
* added tests
* added base implementation for bitrate and threads
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* add migration
* verify running migration populate new value
* implemented service
* generate api
* FE works
* FR Works
* fix test
* fix test fixture
* fix test
* fix test
* consolidate api
* fix test
* added test
* pr feedback
* refactor
* click ont humbnail to show feature selection as well
* Added read-only flag for assets, endpoint to trigger file import vs upload
* updated fixtures with new property
* if upload is 'read-only', ensure there is no existing asset at the designated originalPath
* added test for file import as well as detecting existing image at read-only destination location
* Added storage service test for a case where it should not move read-only assets
* upload doesn't need the read-only flag available, just importing
* default isReadOnly on import endpoint to true
* formatting fixes
* create-asset dto needs isReadOnly, so set it to false by default on create, updated api generation
* updated code to reflect changes in MR
* fixed read stream promise return type
* new index for originalPath, check for existing path on import, reglardless of user, to prevent duplicates
* refactor: import asset
* chore: open api
* chore: tests
* Added externalPath support for individual users, updated UI to allow this to be set by admin
* added missing var for externalPath in ui
* chore: open api
* fix: compilation issues
* fix: server test
* built api, fixed user-response dto to include externalPath
* reverted accidental commit
* bad commit of duplicate externalPath in user response dto
* fixed tests to include externalPath on expected result
* fix: unit tests
* centralized supported filetypes, perform file type checking of asset and sidecar during file import process
* centralized supported filetype check method to keep regex DRY
* fixed typo
* combined migrations into one
* update api
* Removed externalPath from shared-link code, added column to admin user page whether external paths / import is enabled or not
* update mimetype
* Fixed detect correct mimetype
* revert asset-upload config
* reverted domain.constant
* refactor
* fix mime-type issue
* fix format
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* add thumbhash: server generation and web impl
* move logic to infra & use byta in db
* remove unnecesary logs
* update generated API and simplify thumbhash gen
* fix check errors
* removed unnecessary library and css tag
* style edits
* syntax mistake
* update server test, change thumbhash job name
* fix tests
* Update server/src/domain/asset/response-dto/asset-response.dto.ts
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
* add unit test, change migration date
* change to official thumbhash impl
* update call method to not use eval
* "generate missing" looks for thumbhash
* improve queue & improve syntax
* update syntax again
* update tests
* fix thumbhash generation
* consolidate queueing to avoid duplication
* cover all types of incorrect thumbnail cases
* split out jest tasks
* put back thumbnail duration loading for images without thumbhash
* Remove stray package.json
---------
Co-authored-by: Luke McCarthy <mail@lukehmcc.com>
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
Given #2668 introduced support for imagemagick and libraw, this should no
longer be necessary which allow for reduced code footprint and complexity.
Fixes: #2744
* Add on this day
* add query for x year
* dev: add query
* dev: front end
* dev: styling
* styling
* more styling
* add new page
* navigating
* navigate back and forth
* styling
* show gallery
* fix test
* fix test
* show previous and next title
* fix test
* show up down scrolling button
* more styling
* styling
* fix app bar
* fix height of next/previous
* autoplay
* auto play
* refactor
* refactor
* refactor
* show date
* Navigate
* finish
* pr feedback
* refactor microservices to machine-learning
* Update tGithub issue template with correct task syntax
* Added microservices container
* Communicate between service based on queue system
* added dependency
* Fixed problem with having to import BullQueue into the individual service
* Added todo
* refactor server into monorepo with microservices
* refactor database and entity to library
* added simple migration
* Move migrations and database config to library
* Migration works in library
* Cosmetic change in logging message
* added user dto
* Fixed issue with testing not able to find the shared library
* Clean up library mapping path
* Added webp generator to microservices
* Update Github Action build latest
* Fixed issue NPM cannot install due to conflict witl Bull Queue
* format project with prettier
* Modified docker-compose file
* Add GH Action for Staging build:
* Fixed GH action job name
* Modified GH Action to only build & push latest when pushing to main
* Added Test 2e2 Github Action
* Added Test 2e2 Github Action
* Implemented microservice to extract exif
* Added cronjob to scan and generate webp thumbnail at midnight
* Refactor to ireduce hit time to database when running microservices
* Added error handling to asset services that handle read file from disk
* Added video transcoding queue to process one video at a time
* Fixed loading spinner on web while loading covering the info panel
* Add mechanism to show new release announcement to web and mobile app (#209)
* Added changelog page
* Fixed issues based on PR comments
* Fixed issue with video transcoding run on the server
* Change entry point content for backward combatibility when starting up server
* Added announcement box
* Added error handling to failed silently when the app version checking is not able to make the request to GITHUB
* Added new version announcement overlay
* Update message
* Added messages
* Added logic to check and show announcement
* Add method to handle saving new version
* Added button to dimiss the acknowledge message
* Up version for deployment to the app store
* WIP fix user e2e tests
The e2e tests still don't seem to work due to migrations not running.
Changes:
- update user.e2e tests to use new `userService.createUser` method
- fix server `typeorm` command to use ORM config
- update make test-e2e to re-create database volume every time
- add User DTO
- update auth.service and user.service to use User DTO
- update CreateUserDto making optional properties that are optional
* Fix migrations
- add missing `.ts` extension to migrations path
- update user e2e test for the new returned User resource
* Setup e2e testing
* Add user e2e tests
* Rename database host env variable to DB_HOST
* Force push (try to recover DB_HOST env)
* Rename db host env variable to `DB_HOSTNAME`
* Remove unnecessary `initDb` from test-utils
The current database.config is running the migrations:
`migrationsRun: true`
