Jonathan Jogenfors
e6f2bb9f89
fix(server): use extension in originalFileName for libraries ( #8083 )
...
* use file base
* fix: test
* fix: e2e-job tests
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-03-19 23:40:28 -05:00
Jason Rasmussen
40262c30cb
refactor(server): library service ( #8050 )
...
* refactor: library service
* chore: open api
* fix: checks
2024-03-18 16:59:53 -04:00
Jason Rasmussen
07e8f79563
refactor: library e2e ( #7969 )
2024-03-15 09:16:08 -04:00
Mert
a097e903c9
feat(server): OpenTelemetry integration ( #7356 )
...
* wip
* span class decorator
fix typing
* improvements
* noisy postgres logs
formatting
* add source
* strict string comparison
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
* remove debug code
* execution time histogram
* remove prometheus stuff
remove prometheus data
* disable by default
disable nestjs-otel stuff by default
update imports
* re-add postgres instrumentation
formatting
formatting
* refactor: execution time histogram
* decorator alias
* formatting
* keep original method order in filesystem repo
* linting
* enable otel sdk in e2e
* actually enable otel sdk in e2e
* share exclude paths
* formatting
* fix rebase
* more buckets
* add example setup
* add envs
fix
actual fix
* linting
* update comments
* update docker env
* use more specific env
---------
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-03-12 05:19:12 +00:00
Jason Rasmussen
30b0b2474e
refactor: asset e2e ( #7769 )
2024-03-09 12:51:58 -05:00
Jason Rasmussen
89f6190fb0
refactor: search e2e ( #7732 )
2024-03-08 11:20:54 -05:00
Jonathan Jogenfors
4cb0f37918
chore(server): Move library watcher to microservices ( #7533 )
...
* move watcher init to micro
* document watcher recovery
* chore: fix lint
* add try lock
* use global library watch lock
* fix: ensure lock stays on
* fix: mocks
* unit test for library watch lock
* move statement to correct test
* fix: correct return type of try lock
* fix: tests
* add library teardown
* add chokidar error handler
* make event strings an enum
* wait for event refactor
* refactor event type mocks
* expect correct error
* don't release lock in teardown
* chore: lint
* use enum
* fix mock
* fix lint
* fix watcher await
* remove await
* simplify typing
* remove async
* Revert "remove async"
This reverts commit 84ab5abac4
2024-03-07 11:36:53 -06:00
Alex
3da2b05428
chore(server): save original file name with extension ( #7679 )
...
* chore(server): save original file name with extension
* extract extension
* update e2e test
* update e2e test
* download archive
* fix download archive appending name
* pr feedback
* remove unused code
* test
* unit test
* remove unused code
* migration
* noops
* pr feedback
* Update server/src/domain/download/download.service.ts
Co-authored-by: Mert <101130780+mertalev@users.noreply.github.com>
---------
Co-authored-by: Mert <101130780+mertalev@users.noreply.github.com>
2024-03-07 02:34:55 +00:00
Alex
52dfe5fc92
fix(server): stack info in asset response for mobile ( #7346 )
...
* fix(server): stack info in asset response for mobile
* fix(server): getAllAssets - do not filter by stack ID
* tet(server): GET /assets stack e2e
* chore(server): fix checks
* stack asset height
---------
Co-authored-by: shenlong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
2024-03-06 05:44:56 +00:00
Jonathan Jogenfors
5d377e5b0f
chore(server): eslint await-thenable ( #7545 )
...
* await-thenable
* fix library watchers
* moar eslint
* fix test
* fix typo
* try to remove check void return
* fix checksVoidReturn
* move to domain utils
* remove eslint ignores
* chore: cleanup types
* chore: use logger
* fix: e2e
---------
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-03-05 17:23:06 -05:00
Jason Rasmussen
3d25d91e77
refactor: library e2e ( #7538 )
...
* refactor: library e2e
* refactor: remove before each usages
2024-02-29 15:10:08 -05:00
Jonathan Jogenfors
efa6efd200
feat(server,web): remove external path nonsense and make libraries admin-only ( #7237 )
...
* remove external path
* open-api
* make sql
* move library settings to admin panel
* Add documentation
* show external libraries only
* fix library list
* make user library settings look good
* fix test
* fix tests
* fix tests
* can pick user for library
* fix tests
* fix e2e
* chore: make sql
* Use unauth exception
* delete user library list
* cleanup
* fix e2e
* fix await lint
* chore: remove unused code
* chore: cleanup
* revert docs
* fix: is admin stuff
* table alignment
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-02-29 12:35:37 -06:00
Jason Rasmussen
af0de1a768
chore: linting ( #7532 )
...
* chore: linting
* fix: broken tests
* fix: formatting
2024-02-29 11:26:55 -05:00
Jason Rasmussen
bb3d81bfc5
chore: build tweaks ( #7484 )
2024-02-29 09:22:25 -05:00
Jason Rasmussen
74d431f881
refactor(server): format and metadata e2e ( #7477 )
...
* refactor(server): format and metadata e2e
* refactor: on upload success waiting
2024-02-28 10:21:31 +01:00
Jason Rasmussen
807cd245f4
refactor(server): e2e ( #7462 )
...
* refactor: trash e2e
* refactor: asset e2e
2024-02-27 14:04:38 -05:00
Jason Rasmussen
2ebb57cbd4
refactor: e2e client ( #7324 )
2024-02-21 17:34:11 -05:00
Jason Rasmussen
546edc2e91
refactor: album e2e ( #7320 )
...
* refactor: album e2e
* refactor: user e2e
2024-02-21 16:52:13 -05:00
Jason Rasmussen
f798e037d5
refactor(server): e2e ( #7265 )
...
* refactor: activity e2e
* refactor: person e2e
* refactor: shared link e2e
2024-02-21 08:28:03 -05:00
Mert
eb73f6605b
fix(server): don't return archived assets by default ( #7278 )
...
* don't show archived results by default
* fix e2e
* generate sql
* set default in dto
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-02-21 03:59:26 +00:00
Jonathan Jogenfors
b3c7bebbd4
feat(server,web) Semantic import path validation ( #7076 )
...
* add library validation api
* chore: open api
* show warning i UI
* add flex row
* fix e2e
* tests
* fix tests
* enforce path validation
* enforce validation on refresh
* return 400 on bad import path
* add limits to import paths
* set response code to 200
* fix e2e
* fix lint
* fix test
* restore e2e folder
* fix import
* use startsWith
* icon color
* notify user of failed validation
* add parent div to validation
* add docs to the import validation
* improve library troubleshooting docs
* fix button alignment
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-02-20 09:53:12 -06:00
Jason Rasmussen
9b20604a70
refactor(server): e2e ( #7223 )
...
* refactor: download e2e
* refactor: oauth e2e
* refactor: system config e2e
* refactor: partner e2e
* refactor: server-info e2e
* refactor: user e2e
* chore: uncomment test
2024-02-19 22:34:18 -05:00
Jason Rasmussen
a03b37ca86
refactor: server auth e2e ( #7203 )
2024-02-19 12:03:51 -05:00
Michel Heusschen
0730b54ca9
fix(server): empty/restore trash ( #7161 )
...
* fix(server): empty/restore trash
* add e2e tests
* add e2e tests - part 2
2024-02-17 14:05:43 -06:00
Jason Rasmussen
b648025e2f
refactor: library watching ( #7071 )
2024-02-13 08:48:47 -05:00
Mert
e334443919
feat(server, web): smart search filtering and pagination ( #6525 )
...
* initial pagination impl
* use limit + offset instead of take + skip
* wip web pagination
* working infinite scroll
* update api
* formatting
* fix rebase
* search refactor
* re-add runtime config for vector search
* fix rebase
* fixes
* useless omitBy
* unnecessary handling
* add sql decorator for `searchAssets`
* fixed search builder
* fixed sql
* remove mock method
* linting
* fixed pagination
* fixed unit tests
* formatting
* fix e2e tests
* re-flatten search builder
* refactor endpoints
* clean up dto
* refinements
* don't break everything just yet
* update openapi spec & sql
* update api
* linting
* update sql
* fixes
* optimize web code
* fix typing
* add page limit
* make limit based on asset count
* increase limit
* simpler import
2024-02-12 20:50:47 -05:00
martin
f1e4fdf175
refactor: authentication on public routes ( #6765 )
...
* refactor: authentication on public routes
* fix: remove public user
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* remove unused method
* fix: tests
* fix: useless methods
* fix: tests
* pr feedback
* pr feedback
* chore: cleanup
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-02-13 01:47:26 +00:00
Jonathan Jogenfors
954c1c2ef4
feat(server): server-side checking of duplicate import paths and exclusion patterns ( #6993 )
...
validate path and pattern
2024-02-09 13:02:11 -05:00
Jason Rasmussen
90a7f16817
chore(server): remove deprecated endpoints ( #6984 )
...
* chore: remove deprecated endpoints
* chore: open api
2024-02-08 16:57:54 -05:00
renovate[bot]
479fca8f02
chore(deps): pin tensorchord/pgvecto-rs docker tag to 9072418 ( #6960 )
...
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-02-07 02:51:22 +00:00
Mert
56b0643890
feat(server)!: pgvecto.rs 0.2 and pgvector compatibility ( #6785 )
...
* basic changes
update version check
set ef_search for clip
* pgvector compatibility
Revert "pgvector compatibility"
This reverts commit 2b66a52aa4097dd27da58138c5288fd87cb9b24a.
pgvector compatibility: minimal edition
pgvector startup check
* update extension at startup
* wording
shortened vector extension variable name
* nightly docker
* fixed version checks
* update tests
add tests for updating extension
remove unnecessary check
* simplify `getRuntimeConfig`
* wording
* reindex on minor version update
* 0.2 upgrade testing
update prod compose
* acquire lock for init
* wip vector down on shutdown
* use upgrade helper
* update image tag
* refine restart check
check error message
* test reindex
testing
upstream fix
formatting
fixed reindexing
* use enum in signature
* fix tests
remove unused code
* add reindexing tests
* update to official 0.2
remove alpha from version name
* add warning test if restart required
* update test image to 0.2.0
* linting and test cleanup
* formatting
* update sql
* wording
* handle setting search path for new and existing databases
* handle new db in reindex check
* fix post-update reindexing
* get dim size
* formatting
* use vbase
* handle different db name
* update sql
* linting
* fix suggested env
2024-02-06 21:46:38 -05:00
Jason Rasmussen
b0d7434e7b
fix: library watching ( #6802 )
2024-01-31 11:26:51 -05:00
Jonathan Jogenfors
068e703e88
feat(server): Automatic watching of library folders ( #6192 )
...
* feat: initial watch support
* allow offline files
* chore: ignore query errors when resetting e2e db
* revert db query
* add savepoint
* guard the user query
* chore: openapi and db migration
* wip
* support multiple libraries
* fix tests
* wip
* can now cleanup chokidar watchers
* fix unit tests
* add library watch queue
* add missing init from merge
* wip
* can now filter file extensions
* remove watch api from non job client
* Fix e2e test
* watch library with updated import path and exclusion pattern
* add library watch frontend ui
* case sensitive watching extensions
* can auto watch libraries
* move watcher e2e tests to separate file
* don't watch libraries from a queue
* use event emitters
* shorten e2e test timeout
* refactor chokidar code to filesystem provider
* expose chokidar parameters to config file
* fix storage mock
* set default config for library watching
* add fs provider mocks
* cleanup
* add more unit tests for watcher
* chore: fix format + sql
* add more tests
* move unwatch feature back to library service
* add file event unit tests
* chore: formatting
* add documentation
* fix e2e tests
* chore: fix e2e tests
* fix library updating
* test cleanup
* fix typo
* cleanup
* fixing as per pr comments
* reduce library watch config file
* update storage config and mocks
* move negative event tests to unit tests
* fix library watcher e2e
* make watch configuration global
* remove the feature flag
* refactor watcher teardown
* fix microservices init
* centralize asset scan job queue
* improve docs
* add more tests
* chore: open api
* initialize app service
* fix docs
* fix library watch feature flag
* Update docs/docs/features/libraries.md
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
* fix: import right app service
* don't be truthy
* fix test speling
* stricter library update tests
* move fs watcher mock to external file
* subscribe to config changes
* docker does not need polling
* make library watch() private
* feat: add configuration ui
---------
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-31 08:15:54 +00:00
Ben McCann
9c7dee8551
chore: migrate CLI to ESM and vitest ( #6777 )
...
* chore: migrate CLI to ESM and vitest
* fix lint
* update github workflow
* format
2024-01-30 18:23:33 -05:00
martin
1bfef200a5
fix(server): avoid leaking people data on shared links ( #6779 )
...
* fix: avoid leaking people data on shared links
* test: add e2e test
2024-01-30 15:34:00 -06:00
Jason Rasmussen
6dca47c629
refactor: no experimental vm modules ( #6719 )
2024-01-29 09:11:02 -06:00
Mert
ae7f174948
refactor: rename clip -> smart search ( #6713 )
2024-01-29 09:51:22 -05:00
martin
fa0913120d
feat(web,server): search people ( #5703 )
...
* feat: search peoples
* fix: responsive design
* use existing count
* generate sql file
* fix: tests
* remove visible people
* fix: merge, hide...
* use component
* fix: linter
* chore: regenerate api
* fix: change name when searching for a face
* save search
* remove duplicate
* use enums for query parameters
* fix: increase to 20 for the local search
* use constants
* simplify
* fix: number of people more visible
* fix: merge
* fix: search
* fix: loading spinner position
* pr feedback
2024-01-27 19:54:31 -05:00
Zack Pollard
25cad79657
refactor: move asset stacks to their own entity ( #6353 )
...
* feat: auto-stack burst photos
* feat: move stacks to asset stack entity
* chore: pin node version with volta in server
* chore: update e2e cases
* chore: cleanup
* feat: migrate existing stacks
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-27 18:52:14 +00:00
Ben McCann
5cec8d1c63
chore: remove unused test util methods ( #6689 )
...
* chore: remove unused test util methods
* format
2024-01-27 12:27:55 -05:00
Jason Rasmussen
b306cf564e
refactor(server): move asset detail endpoint to new controller ( #6636 )
...
* refactor(server): move asset by id to new controller
* chore: open api
* refactor: more consolidation
* refactor: asset service
2024-01-25 12:52:21 -05:00
Mert
bd87eb309c
feat(server): optimize partial facial recognition ( #6634 )
...
* optimize partial facial recognition
* add tests
* use map
* bulk insert faces
2024-01-25 01:27:39 -05:00
Jason Rasmussen
852effa998
refactor(server): e2e ( #6632 )
2024-01-24 17:24:53 -05:00
Aram Akhavan
a972dd4060
fix(server): extraction of Samsung Motionphoto videos ( #6337 )
...
* Fix extraction of samsung motionphoto videos
* Refactor binary tag extraction to the repository to consolidate exiftool usage
* format
* fix linting and swap argument orders
* Fix tag name and conditional order
* Add unit test
* Update server test assets submodule
* Remove old motion photo video assets when a new one is extracted
* delete first, then write
* Include motion photo asset uuid's in the filename
If the filenames are not uniquified, then we can't delete old/corrupt ones
* Fix formatting and fix/add tests
* chore: only use new uuid
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-22 13:04:45 -05:00
Mert
68f52818ae
feat(server): separate face clustering job ( #5598 )
...
* separate facial clustering job
* update api
* fixed some tests
* invert clustering
* hdbscan
* update api
* remove commented code
* wip dbscan
* cleanup
removed cluster endpoint
remove commented code
* fixes
updated tests
minor fixes and formatting
fixed queuing
refinements
* scale search range based on library size
* defer non-core faces
* optimizations
removed unused query option
* assign faces individually for correctness
fixed unit tests
remove unused method
* don't select face embedding
update sql
linting
fixed ml typing
* updated job mock
* paginate people query
* select face embeddings because typeorm
* fix setting face detection concurrency
* update sql
formatting
linting
* simplify logic
remove unused imports
* more specific delete signature
* more accurate typing for face stubs
* add migration
formatting
* chore: better typing
* don't select embedding by default
remove unused import
* updated sql
* use normal try/catch
* stricter concurrency typing and enforcement
* update api
* update job concurrency panel to show disabled queues
formatting
* check jobId in queueAll
fix tests
* remove outdated comment
* better facial recognition icon
* wording
wording
formatting
* fixed tests
* fix
* formatting & sql
* try to fix sql check
* more detailed description
* update sql
* formatting
* wording
* update `minFaces` description
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-18 00:08:48 -05:00
Jason Rasmussen
6f291006e4
fix(server): handle 5 digit years ( #6457 )
2024-01-17 16:08:38 -05:00
Sushain Cherivirala
7fc1954e2a
fix(server): add filename search ( #6394 )
...
Fixes https://github.com/immich-app/immich/issues/5982 .
There are basically three options:
1. Search `originalFileName` by dropping a file extension from the query
(if present). Lower fidelity but very easy - just a standard index &
equality.
2. Search `originalPath` by adding an index on `reverse(originalPath)`
and using `starts_with(reverse(query) + "/", reverse(originalPath)`. A
weird index & query but high fidelity.
3. Add a new generated column called `originalFileNameWithExtension` or
something. More storage, kinda jank.
TBH, I think (1) is good enough and easy to make better in the future.
For example, if I search "DSC_4242.jpg", I don't really think it matters
if "DSC_4242.mov" also shows up.
edit: There's a fourth approach that we discussed a bit in Discord and
decided we could switch to it in the future: using a GIN. The minor
issue is that Postgres doesn't tokenize paths in a useful (they're a
single token and it won't match against partial components). We can
solve that by tokenizing it ourselves. For example:
```
immich=# with vecs as (select to_tsvector('simple', array_to_string(string_to_array('upload/library/sushain/2015/2015-08-09/IMG_275.JPG', '/'), ' ')) as vec) select * from vecs where vec @@ phraseto_tsquery('simple', array_to_string(string_to_array('library/sushain', '/'), ' '));
vec
-------------------------------------------------------------------------------
'-08':6 '-09':7 '2015':4,5 'img_275.jpg':8 'library':2 'sushain':3 'upload':1
(1 row)
```
The query is also tokenized with the 'split-by-slash-join-with-space'
strategy. This strategy results in `IMG_275.JPG`, `2015`, `sushain` and
`library/sushain` matching. But, `08` and `IMG_275` do not match. The
former is because the token is `-08` and the latter because the
`img_275.jpg` token is matched against exactly.
2024-01-15 14:40:28 -06:00
Tom Vincent
984feafb90
fix(server): extract image description ( #6344 )
2024-01-15 11:19:41 -05:00
cfitzw
deb1f970a8
feat(server, web): quotas ( #4471 )
...
* feat: quotas
* chore: open api
* chore: update status box and upload error message
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-12 19:43:36 -05:00
Jason Rasmussen
bf1dd36fa9
refactor(server): split api and jobs into separate e2e suites ( #6307 )
...
* refactor: domain and infra modules
* refactor(server): e2e tests
2024-01-09 23:04:16 -05:00
