* fix: pgvecto.rs extension breaks typeorm schema:drop command
* fix: parse postgres bigints to javascript number types when selecting data
* feat: verify file size is the same as original asset after copying file for storage template job
* feat: allow disabling of storage template job, defaults to disabled for new instances
* fix: don't allow setting concurrency for storage template migration, can cause race conditions above 1
* feat: add checksum verification when file is copied for storage template job
* fix: extract metadata for assets that aren't visible on timeline
* feat: add system metadata repository for storing key values for internal usage
* feat: add database entities for geodata
* feat: move reverse geocoding from local-reverse-geocoder to postgresql
* infra: disable synchronization for geodata_places table until typeorm supports earth column
* feat: remove cities override config as we will default all instances to cities500 now
* test: e2e tests don't clear geodata tables on reset
* WIP: Added immich cli tool to `immich-server` image
* WIP: Added doc entry to show it is preinstalled
* WIP: Moved immich upload cli to `immich` and default to `immich-admin`
* WIP: undid previous commit
* WIP: Updated server docs with new `immich-admin` command
* Added read-only flag for assets, endpoint to trigger file import vs upload
* updated fixtures with new property
* if upload is 'read-only', ensure there is no existing asset at the designated originalPath
* added test for file import as well as detecting existing image at read-only destination location
* Added storage service test for a case where it should not move read-only assets
* upload doesn't need the read-only flag available, just importing
* default isReadOnly on import endpoint to true
* formatting fixes
* create-asset dto needs isReadOnly, so set it to false by default on create, updated api generation
* updated code to reflect changes in MR
* fixed read stream promise return type
* new index for originalPath, check for existing path on import, reglardless of user, to prevent duplicates
* refactor: import asset
* chore: open api
* chore: tests
* Added externalPath support for individual users, updated UI to allow this to be set by admin
* added missing var for externalPath in ui
* chore: open api
* fix: compilation issues
* fix: server test
* built api, fixed user-response dto to include externalPath
* reverted accidental commit
* bad commit of duplicate externalPath in user response dto
* fixed tests to include externalPath on expected result
* fix: unit tests
* centralized supported filetypes, perform file type checking of asset and sidecar during file import process
* centralized supported filetype check method to keep regex DRY
* fixed typo
* combined migrations into one
* update api
* Removed externalPath from shared-link code, added column to admin user page whether external paths / import is enabled or not
* update mimetype
* Fixed detect correct mimetype
* revert asset-upload config
* reverted domain.constant
* refactor
* fix mime-type issue
* fix format
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* add thumbhash: server generation and web impl
* move logic to infra & use byta in db
* remove unnecesary logs
* update generated API and simplify thumbhash gen
* fix check errors
* removed unnecessary library and css tag
* style edits
* syntax mistake
* update server test, change thumbhash job name
* fix tests
* Update server/src/domain/asset/response-dto/asset-response.dto.ts
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
* add unit test, change migration date
* change to official thumbhash impl
* update call method to not use eval
* "generate missing" looks for thumbhash
* improve queue & improve syntax
* update syntax again
* update tests
* fix thumbhash generation
* consolidate queueing to avoid duplication
* cover all types of incorrect thumbnail cases
* split out jest tasks
* put back thumbnail duration loading for images without thumbhash
* Remove stray package.json
---------
Co-authored-by: Luke McCarthy <mail@lukehmcc.com>
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* use imagemagick and libraw for raw image support
imagemagick and libraw have generally good support for raw images, including
Sony's ARW format. These tools should also allow Immich to support many more
image formats in future without any major code changes.
https://www.libraw.org/supported-cameras
I've tested and verified this change with .ARW files and other standard formats.
Fixes: #2156
* Add additional type for awr
* pr feedback
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* feat(server): dynamic job concurrency
* styling and add setting info to top of the job list
* regenerate api
* remove DETECT_OBJECT job
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* chore: update to node 18 and alpine 3.17
Signed-off-by: martin <martin.labat92@gmail.com>
* chore: fix sharp version
Signed-off-by: martin <martin.labat92@gmail.com>
* chore(server): use vips-dev
Signed-off-by: martin <martin.labat92@gmail.com>
* update checkDiskUsage
Signed-off-by: martin <martin.labat92@gmail.com>
* fix: use vips-heif instead of libheif
Signed-off-by: martin <martin.labat92@gmail.com>
* fix: use vips instead of vips-cpp
Signed-off-by: martin <martin.labat92@gmail.com>
* fix: ensure vips installation
Signed-off-by: martin <martin.labat92@gmail.com>
---------
Signed-off-by: martin <martin.labat92@gmail.com>
* chore(server): remove unused device info code
* chore: generate open api
* remove any DeviceTypeEnum usage from mobile
* chore: coverage
* fix: drop device info table
---------
Co-authored-by: Fynn Petersen-Frey <zody22@gmail.com>
* feat: manage authorized devices
* chore: open api
* get header from mobile app
* write header from mobile app
* styling
* fix unit test
* feat: use relative time
* feat: update access time
* fix: tests
* chore: confirm wording
* chore: bump test coverage thresholds
* feat: add some icons
* chore: icon tweaks
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* build: add typesense to docker
* feat(server): typesense search
* feat(web): search
* fix(web): show api error response message
* chore: search tests
* chore: regenerate open api
* fix: disable typesense on e2e
* fix: number properties for open api (dart)
* fix: e2e test
* fix: change lat/lng from floats to typesense geopoint
* dev: Add smartInfo relation to findAssetById to be able to query against it
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* infra: make api-key primary key column a UUID
* infra: move ManyToMany relations in album entity, make ownerId ManyToOne
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* fix(web/server): Uploaded asset to shared link does not get added to the shared link/album
* remove unused code
* Add endpoints for each remove and add assets to shared link
* Update api
* Added deletion logic
* Convert callback to async/await
* Fix linter
* Fix test
* Fix server test
* added test
* Test coverage
* modify DTO
* Add notification
* fix test
* chore(mobile): invoke logout() on mobile app
* feat: add mechanism to delete token from logging out endpoint
* fix: set state after login sequence success
* fix: not removing token when logging out from OAuth
* fix: prettier
* refactor: using accessTokenId to delete
* chore: pr comments
* fix: test
* fix: test threshold
* chore: remove @tensorflow/tfjs-node-gpu as it is unused
* chore: remove ffmpeg from machine-learning docker image
* chore: remove unneeded dependencies + move dev dependencies in server
* chore: reduce server image size
* chore: machine-learning remove extraneous dependencies
* chore: web remove extraneous dependencies
* chore: web Dockerfile reduce production image size
* chore: add exiftool-vendored.pl as a dependency
* chore: add typeorm commands to npm and set default database config values
* feat: move to server side authentication tokens
* fix: websocket should emit error and disconnect on error thrown by the server
* refactor: rename cookie-auth-strategy to user-auth-strategy
* feat: user tokens and API keys now use SHA256 hash for performance improvements
* test: album e2e test remove unneeded module import
* infra: truncate api key table as old keys will no longer work with new hash algorithm
* fix(server): e2e tests (#1435)
* fix: root module paths
* chore: linting
* chore: rename user-auth to strategy.ts and make validate return AuthUserDto
* fix: we should always send HttpOnly for our auth cookies
* chore: remove now unused crypto functions and jwt dependencies
* fix: return the extra fields for AuthUserDto in auth service validate
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* Trying to get exifdata working with different lib.
* Got the new library working.
* Addressing PR comments.
* Removed not used vars and proper place for the eslint disable.
* Fix time-utils to use the exiftool-vendored lib.
Fixed also one test, as that would be valid.
* Using filename for timestamp as well if possible.
* Add new tests for time-utils.
* Remember to gracefully terminate the exiftool instance when not needed.
* eslint ignore...
* Apperantly Dockerfile changes were not pushed.
* feat(dockerfile): Tweak the Server Dockerfile
* feat(server): getTimestampFromFilename should return string or undefined.
* feat(server): If we don't have exifData or timestamp from filename, raise an error.
* Apparently test was already right, but my local system disagrees.
* More utilities for parsing and fix the timestampFromFilename.
It was returning an incorrect date as the regex doesn't seem to be the best for this as files named `IMG_0115.HEIC` will want to get parsed incorrectly due to it.
* feat(server/docker): Install perl as it seems to be required.
* feat(server): remember to include exposureTime and focalLength in new exif data.
* feat(server): Remove the parsing from filename as requested.
* feat(server): Import exiftool differently in time-utils.
* feat(server): Error handling when there is no exifData.
* feat(server): Fixes for the error handling when there is no exifData.
* feat(server): Remember to include modifyDate despite no exif.
* feat(server): Remember to include model of Camera.
* feat(server): Fixing up Exiftool usage.
Including proper logging for it, which had to be done in wrapped fashion due to it expecting all the logging levels which NextJS logger doesn't implement.
* feat(server): Do not use a wrapper for ExifTool logging.
* fix merge conflicts in metadata-extractor
* chore(web,server): run code coverage reports
* chore(tests): fail test check if coverage drops
* chore: disable e2e until they are fixed
* chore(web): coverage threshold
* refactor: user repository
* refactor: user module
* refactor: move database into infra
* refactor(cli): use user core
* chore: import path
* chore: tests
- Refactor user business logic from `user.service` into `user.domain`
Make user business logic reusable by using `user.domain` from other services than `user.service`
- Add `jest-when` lib to make testing easier and use it in `userService`
Using when helps from coupling tests to order of mock implementations execution
- Move all user business logic from user-repository to user.service
- Fix user.service tests not awaiting promises leaking state between tests
- Presentation logic for `getUserProfileImage` moved from UserService to UserController
- Fix `user.e2e` test logic. Pending fixing the configuration of the test itself
* feat: add admin config module for user configured config, uses it for ffmpeg
* feat: add api endpoint to retrieve admin config settings and values
* feat: add settings panel to admin page on web (wip)
* feat: add api endpoint to update the admin config
* chore: re-generate openapi spec after rebase
* refactor: move from admin config to system config naming
* chore: move away from UseGuards to new @Authenticated decorator
* style: dark mode styling for lists and fix conflicting colors
* wip: 2 column design, no edit button
* refactor: system config
* chore: generate open api
* chore: rm broken test
* chore: cleanup types
* refactor: config module names
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack.pollard@moonpig.com>
Implemented a mechanism to extract the correct time zone from the GPS coordinate if presented in the file's EXIF, and to convert the timestamp to the correct UTC time so that the time will show correctly based on the mobile/web local time zone.
* Added swagger bearer auth method authentication accordingly
* Update Auth endpoint
* Added additional api information for authentication
* Added Swagger CLI pluggin
* Added DTO for /user endpoint
* Added /device-info reponse DTOs
* Implement server version
* Added DTOs for /server-info
* Added DTOs for /assets
* Added album to Swagger group
* Added generated specs file
* Add Client API generator for web
* Remove incorrectly placed node_modules
* Created class to handle access token
* Remove password and hash when getting all user
* PR feedback
* Fixed video from CLI doesn't get metadata extracted
* Fixed issue with TSConfig to work with generated openAPI
* PR feedback
* Remove console.log
* fix: remove config parameter from typeorm cli and update config
the config parameter is no longer supported since version 0.3
the config now needs to export a DataSource object to work with the 0.3 cli
* fix: update all typeorm entities and migrations to be aligned with database structure
* Fixed test-util import databaseConfig
* Fixed column mismatch in raw query with new migration
* Remove dist build directory when starting dev server
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* Remove thumbnail generation on mobile
* Remove tconditions for missing thumbnail on the backend
* Remove console.log
* Refactor queue systems
* Convert queue and processor name to constant
* Added corresponding interface to job queue
* Fix lint issues and some other TS issues
- set TypeScript in strict mode
- add npm commands to lint / check code
- fix all lint issues
- fix some TS issues
- rename User reponse DTO to make it consistent with the other ones
- override Express/User interface to use UserResponseDto interface
This is for when the accessing the `user` from a Express Request,
like in `asset-upload-config`
* Fix the rest of TS issues
- fix all the remaining TypeScript errors
- add missing `@types/mapbox__mapbox-sdk` package
* Move global.d.ts to server `src` folder
* Update AssetReponseDto duration type
This is now of type `string` that defaults to '0:00:00.00000' if not set
which is what the mobile app currently expects
* Set context when logging error in asset.service
Use `ServeFile` as the context for logging an error when
asset.resizePath is not set
* Fix wrong AppController merge conflict resolution
`redirectToWebpage` was removed in main as is no longer used.
* Added file selector
* Extract metadata to upload files to the web
* Added request for uploading
* Generate jpeg/Webp thumbnail for asset uploaded without thumbnail data
* Added generating thumbnail for video and WebSocket broadcast after thumbnail is generated
* Added video length extraction
* Added Uploading Panel
* Added upload progress store and styling the uploaded asset
* Added condition to only show upload panel when there is upload in progress
* Remove asset from the upload list after successfully uploading
* Added WebSocket to listen to upload event on the web
* Added mechanism to check for existing assets before uploading on the web
* Added test workflow
* Update readme
* Rename "shared" to "album"
Prepare moving "SharedAlbums" to "Albums"
* Update server album API endpoints
* Update mobile app album endpoints
Also add `putRequest` to mobile network.service
* Add GET album collection filter
- allow to filter by owner = 'mine' | 'their'
- make sharedWithUserIds no longer required when creating an album
* Rename remaining variables to "album"
* Add ParseMeUUIDPipe to validate uuid or `me`
* Add album params validation
* Update todo in mobile album service.
* Setup e2e testing
* Add user e2e tests
* Rename database host env variable to DB_HOST
* Add some `Album` e2e tests
Also fix issues found with the tests
* Force push (try to recover DB_HOST env)
* Rename db host env variable to `DB_HOSTNAME`
* Remove unnecessary `initDb` from test-utils
The current database.config is running the migrations:
`migrationsRun: true`
* Remove `initDb` usage from album e2e test
* Update GET albums filter to `shared`
- add filter by all / shared / not shared
- add response DTOs
- add GET albums e2e tests
* Update album e2e tests for user.service changes
* Update mobile app to use album response DTOs
* Refactor album-service DB into album-registry
- DB logic refactored into album-repository making it easier to test
- add some album-service unit tests
- add `clearMocks` to jest configuration
* Finish implementing album.service unit tests
* Rename response DTO
Make them consistent with rest of the project naming
* Update debug log messages in mobile network service
* Rename table `shared_albums` to `albums`
* Rename table `asset_shared_album`
* Rename Albums `sharedAssets` to `assets`
* Update tests to match updated "delete" response
* Fixed asset cannot be compared in Set by adding Equatable package
* Remove hero effect to fixed janky animation
Co-authored-by: Alex <alex.tran1502@gmail.com>
* refactor microservices to machine-learning
* Update tGithub issue template with correct task syntax
* Added microservices container
* Communicate between service based on queue system
* added dependency
* Fixed problem with having to import BullQueue into the individual service
* Added todo
* refactor server into monorepo with microservices
* refactor database and entity to library
* added simple migration
* Move migrations and database config to library
* Migration works in library
* Cosmetic change in logging message
* added user dto
* Fixed issue with testing not able to find the shared library
* Clean up library mapping path
* Added webp generator to microservices
* Update Github Action build latest
* Fixed issue NPM cannot install due to conflict witl Bull Queue
* format project with prettier
* Modified docker-compose file
* Add GH Action for Staging build:
* Fixed GH action job name
* Modified GH Action to only build & push latest when pushing to main
* Added Test 2e2 Github Action
* Added Test 2e2 Github Action
* Implemented microservice to extract exif
* Added cronjob to scan and generate webp thumbnail at midnight
* Refactor to ireduce hit time to database when running microservices
* Added error handling to asset services that handle read file from disk
* Added video transcoding queue to process one video at a time
* Fixed loading spinner on web while loading covering the info panel
* Add mechanism to show new release announcement to web and mobile app (#209)
* Added changelog page
* Fixed issues based on PR comments
* Fixed issue with video transcoding run on the server
* Change entry point content for backward combatibility when starting up server
* Added announcement box
* Added error handling to failed silently when the app version checking is not able to make the request to GITHUB
* Added new version announcement overlay
* Update message
* Added messages
* Added logic to check and show announcement
* Add method to handle saving new version
* Added button to dimiss the acknowledge message
* Up version for deployment to the app store
* WIP fix user e2e tests
The e2e tests still don't seem to work due to migrations not running.
Changes:
- update user.e2e tests to use new `userService.createUser` method
- fix server `typeorm` command to use ORM config
- update make test-e2e to re-create database volume every time
- add User DTO
- update auth.service and user.service to use User DTO
- update CreateUserDto making optional properties that are optional
* Fix migrations
- add missing `.ts` extension to migrations path
- update user e2e test for the new returned User resource
* Update readme
* Added webp to table and entity
* Added cronjob and sharp dependencies
* Added conversion of webp every 5 minutes and endpoint will now server webp image if exist
* Refactor docker-compose to its own folder
* Added FastAPI development environment
* Added support for GPU in docker file
* Added image classification
* creating endpoint for smart Image info
* added logo with white background on ios
* Added endpoint and trigger for image tagging
* Classify image and save into database
* Update readme
* Support HEIC/HEIF backup
* Storing backup directly from original file from the phone
* Directly read and backup video file - Improve performance on video backup
* Implementing video upload features
* setup image resize processor
* Add video thumbnail with duration and icon
* Fixed issue with video upload timeout and upper case file type on ios
* Added video player page
* Added video player page
* Fixing video player not play on ios
* Added partial file streaming for ios/android video request
* Added nginx as proxy server for better file serving
* update nginx and docker-compose file
* Video player working correctly
* Video player working correctly
* Split duration to the second
