feat(server): better mount checks (#13092)

docs/docs/administration/system-integrity.md

@@ -0,0 +1,47 @@
+# System Integrity
+
+## Folder checks
+
+:::info
+The folders considered for these checks include: `upload/`, `library/`, `thumbs/`, `encoded-video/`, `profile/`
+:::
+
+When Immich starts, it performs a series of checks in order to validate that it can read and write files to the volume mounts used by the storage system. If it cannot perform all the required operations, it will fail to start. The checks include:
+
+- Creating an initial hidden file (`.immich`) in each folder
+- Reading a hidden file (`.immich`) in each folder
+- Overwriting a hidden file (`.immich`) in each folder
+
+The checks are designed to catch the following situations:
+
+- Incorrect permissions (cannot read/write files)
+- Missing volume mount (`.immich` files should exist, but are missing)
+
+### Common issues
+
+:::note
+`.immich` files serve as markers and help keep track of volume mounts being used by Immich. Except for the situations listed below, they should never be manually created or deleted.
+:::
+
+#### Missing `.immich` files
+
+```
+Verifying system mount folder checks (enabled=true)
+...
+ENOENT: no such file or directory, open 'upload/encoded-video/.immich'
+```
+
+The above error messages show that the server has previously (successfully) written `.immich` files to each folder, but now does not detect them. This could be because any of the following:
+
+- Permission error - unable to read the file, but it exists
+- File does not exist - volume mount has changed and should be corrected
+- File does not exist - user manually deleted it and should be manually re-created (`touch .immich`)
+- File does not exist - user restored from a backup, but did not restore each folder (user should restore all folders or manually create `.immich` in any missing folders)
+
+### Ignoring the checks
+
+The checks are designed to catch common problems that we have seen users have in the past, but if you want to disable them you can set the following environment variable:
+
+```
+IMMICH_IGNORE_MOUNT_CHECK_ERRORS=true
+```

docs/docs/install/environment-variables.md

@@ -42,6 +42,7 @@ These environment variables are used by the `docker-compose.yml` file and do **N
 | `IMMICH_MICROSERVICES_METRICS_PORT` | Port for the OTEL metrics                                                                 |            `8082`            | server                   | microservices      |
 | `IMMICH_PROCESS_INVALID_IMAGES`     | When `true`, generate thumbnails for invalid images                                       |                              | server                   | microservices      |
 | `IMMICH_TRUSTED_PROXIES`            | List of comma separated IPs set as trusted proxies                                        |                              | server                   | api                |
+| `IMMICH_IGNORE_MOUNT_CHECK_ERRORS`  | See [System Integrity](/docs/administration/system-integrity)                             |                              | server                   | api, microservices |
 
 \*1: `TZ` should be set to a `TZ identifier` from [this list][tz-list]. For example, `TZ="Etc/UTC"`.
 `TZ` is used by `exiftool` as a fallback in case the timezone cannot be determined from the image metadata. It is also used for logfile timestamps and cron job execution.