mirror of
https://github.com/vcmi/vcmi.git
synced 2026-06-19 22:57:37 +02:00
Merge pull request #7145 from IvanSavenko/server_update
Update docs to account for builds migration
This commit is contained in:
Ivan Savenko
@@ -1,12 +1,36 @@
|
||||
# Builds downloads server
|
||||
|
||||
RAM requirements: non-existing
|
||||
CPU requirements: non-existing
|
||||
SSD requirements: 100 GB, ideally - 200 GB. Can be located on a separate volume
|
||||
- RAM requirements: non-existing
|
||||
- CPU requirements: non-existing
|
||||
- SSD requirements: 100 GB+, can be located on a separate volume
|
||||
|
||||
For storage we need around 50 Gb to store 1 month worth of builds. Actual amount that we can store varies with activity - each accepted PR means one more build to store. Amount that we want to store also depends - they are rarely needed, but we had several cases where we had to track faulty build, so amount nedeed is maximum time between two major releases (up to ~6 months)
|
||||
|
||||
## Configuration
|
||||
|
||||
Contains two users:
|
||||
|
||||
- `uploader`: can only use sftp uploads. Private key for login is stored as Github Secret in vcmi/vcmi. Also available in `/root/.ssh/uploader` on server
|
||||
- `downloader`: used as owner for files available to download for public
|
||||
|
||||
Important file locations:
|
||||
|
||||
- `/opt/vcmiscripts/` - contains scripts that are also available in `scripts` directory next to docs
|
||||
- `/home/uploader/upload` - incoming directory for files uploaded from Github Actions
|
||||
- `/home/downloader/tmp` - temporary storage for files that have finished upload but yet to be sorted. Generally this directory should be empty except for tiny moments of time while script is running OR if script fails to sort some builds such as builds for new platforms with unexpected file names
|
||||
- `/home/downloader/www` - permanent storage for files available to download via nginx by players
|
||||
|
||||
Custom scripts (located in `/opt/vcmiscripts/` directory):
|
||||
|
||||
- `on_cron_update.sh` - runs every minute via cron as root and does following actions:
|
||||
- moves builds that finished upload from `/home/uploader/upload` to `/home/downloader/tmp`, changing file owners and permissions
|
||||
- executes scripts `ensure_free_space.sh` and `sort_builds.sh` as user `downloader`
|
||||
- `ensure_free_space.sh` - automatically removes oldest builds from `/home/downloader/www/`, except for releases in `/home/downloader/www/branch/master`.
|
||||
- `sort_builds.sh` - renames and moves builds from `/home/downloader/tmp` to various directories in `/home/downloader/www/`
|
||||
|
||||
## Setup
|
||||
|
||||
- Copy scripts from `scripts` directory to `/root/scripts`
|
||||
- Copy scripts from `scripts` directory to `/opt/vcmiscripts`
|
||||
- Execute script `setup_uploader.sh`
|
||||
- If needed, attach volume for builds storage via fstab: `/dev/disk/by-id/scsi-0DO_Volume_vcmi-second-builds /home/downloader ext4 defaults,nofail,discard 0 0`
|
||||
- Ensure that `/home/downloader` and `/home/downloader/www` are owned by root, while all `/home/downloader/www` content is owned by `downloader` user
|
||||
@@ -14,3 +38,14 @@ SSD requirements: 100 GB, ideally - 200 GB. Can be located on a separate volume
|
||||
- Add certificates to nginx
|
||||
- Install `fancy_index` nginx module: `apt install libnginx-mod-http-fancyindex`
|
||||
- Restart nginx
|
||||
|
||||
## Migration
|
||||
|
||||
- Detach volume from old server on Digital Ocean control panel
|
||||
- Attach volume to new server
|
||||
- Setup new server as described in this document
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Run script `/opt/vcmiscripts/on_cron_update.sh` manually to check for possible issues
|
||||
- Ensure that command `sudo -u downloader df /home/downloader/www/branch/` works and outputs size of volume with builds. If it results in "permission denied", make sure that all directories in that path have `+x` permission
|
||||
|
||||
@@ -79,13 +79,13 @@ All droplets can only be accessed using ssh login with public key. Currently acc
|
||||
| ------- | -------------- | -------- |
|
||||
| `vcmi-artifactory` | 4 Gb / 2 CPU / 80 Gb / $24 | [Conan Artifactory server](https://artifactory.vcmi.eu/) (WIP) |
|
||||
| `vcmi-forum` | 2 Gb / 1 CPU / 25 Gb / $12 (+20%) | [Discourse forum](https://forum.vcmi.eu/) |
|
||||
| `vcmi-second` | 1 Gb / 1 CPU / 20 Gb + 100 Gb / $6 + $10 | Multiplayer lobby (lobby.vcmi.eu or beholder.vcmi.eu - deprecated). Floating IP: 67.207.75.182, Builds uploading from Github, [Build download page](http://download.vcmi.eu/), [Legacy download page](https://builds.vcmi.download/) |
|
||||
| `vcmi-web` | 512 Mb / 1 CPU / 10 Gb / $4 (+20%) | Contains nginx server for redirecting [old bug tracker](https://bugs.vcmi.eu/), [old wiki](https://wiki.vcmi.eu/), and [old slack invite page](https://slack.vcmi.eu/) |
|
||||
| `vcmi-second` | 1 Gb / 1 CPU / 20 Gb / $6 | Multiplayer lobby (lobby.vcmi.eu or beholder.vcmi.eu - deprecated). Floating IP: 67.207.75.182 |
|
||||
| `vcmi-web` | 512 Mb / 1 CPU / 10 Gb + 100 Gb / $4 (+20%) + $10 | Builds uploading from Github, [Build download page](http://download.vcmi.eu/), [Legacy download page](https://builds.vcmi.download/). Also contains nginx server for redirecting [old bug tracker](https://bugs.vcmi.eu/), [old wiki](https://wiki.vcmi.eu/), and [old slack invite page](https://slack.vcmi.eu/) |
|
||||
|
||||
Notes:
|
||||
|
||||
- Droplets with deployed services have backups enabled (+20% costs)
|
||||
- In addition to droplets, we have separate 100 Gb volume for builds ($10 / month), currently attached to `vcmi-second`
|
||||
- In addition to droplets, we have separate 100 Gb volume for builds ($10 / month), currently attached to `vcmi-web`
|
||||
- There is snapshot for old `vcmi-main` droplet, preserved in case if we need to retrieve some data from it - old bugtracker, forum, and wiki ($1.5 / month)
|
||||
|
||||
### Rules to stick to
|
||||
@@ -111,17 +111,16 @@ If new services added firewall rules can be adjusted in [DO control panel](https
|
||||
| Domain | Content | Hosted on | Notes |
|
||||
| ------ | ------- | --------- | ----- |
|
||||
| [vcmi.eu](https://vcmi.eu) | Main page redirect | CNAME | No content, redirects to [real main page](https://vcmi.github.io/) |
|
||||
| [download.vcmi.eu](https://download.vcmi.eu) | Public downloads & daily builds | `vcmi-second` | - |
|
||||
| [beholder.vcmi.eu](https://beholder.vcmi.eu) | Multiplayer lobby | `vcmi-second` | No http services. Used for VCMI 1.7 lobby and older. Also handles build uploads from Github. Deprecated in favor of lobby and uploads |
|
||||
| [download.vcmi.eu](https://download.vcmi.eu) | Public downloads & daily builds | `vcmi-web` | - |
|
||||
| [upload.vcmi.eu](https://upload.vcmi.eu) | Domain name for uploading daily builds from Github | `vcmi-web` | No http services |
|
||||
| [beholder.vcmi.eu](https://beholder.vcmi.eu) | Multiplayer lobby | `vcmi-second` | No http services. Used for VCMI 1.7.3 lobby and older. Deprecated in favor of `lobby` |
|
||||
| [lobby.vcmi.eu](https://lobby.vcmi.eu) | Multiplayer lobby | `vcmi-second` | No http services |
|
||||
| [forum.vcmi.eu](https://forum.vcmi.eu) | Discourse forum | `vcmi-forum` | - |
|
||||
| [bugs.vcmi.eu](https://bugs.vcmi.eu) | Bug tracker | `vcmi-web` | Redirects to [Github Issues](https://github.com/vcmi/vcmi/issues) |
|
||||
| [slack.vcmi.eu](https://slack.vcmi.eu) | Slack invite page | `vcmi-web` | Redirects to [main page](https://vcmi.eu/) |
|
||||
| [wiki.vcmi.eu](https://wiki.vcmi.eu) | Wiki | `vcmi-web` | Redirects to [main page](https://vcmi.eu/) |
|
||||
| [vcmi.download](https://vcmi.download) | Main page redirect | CNAME | No content, redirects to [main page](https://vcmi.eu/) |
|
||||
| [builds.vcmi.download](https://builds.vcmi.download) | Public downloads | `vcmi-second` | Same content as main download server. Known bug: excessive caching. Potentially deprecated in favor of main domain |
|
||||
| [dependencies.vcmi.download](https://dependencies.vcmi.download) | — | — | Not configured |
|
||||
| [mods.vcmi.download](https://mods.vcmi.download) | — | — | Not configured |
|
||||
| [builds.vcmi.download](https://builds.vcmi.download) | Public downloads | `vcmi-second` | Redirects to [download.vcmi.eu](https://download.vcmi.eu) |
|
||||
|
||||
## Self-hosted services
|
||||
|
||||
|
||||
@@ -8,6 +8,6 @@
|
||||
<a href="https://github.com/vcmi/vcmi">GitHub</a>
|
||||
</h4>
|
||||
<h5>
|
||||
Hosting provided by <a href="https://www.digitalocean.com/" target="_blank" rel="noopener">DigitalOcean</a>
|
||||
<a href="https://m.do.co/c/b2d5cf698ed1"><img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/PoweredByDO/DO_Powered_by_Badge_black.svg" width="201px"></a>
|
||||
</h5>
|
||||
</body></html>
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# Base directory that needs free space
|
||||
TARGET_DIR="/home/downloader/www/branch"
|
||||
# Exlude master builds (releases) from auto-removal
|
||||
EXCLUDE_DIR="/home/downloader/www/branch/master"
|
||||
EXCLUDE_DIR="/home/downloader/www/branch/master/*"
|
||||
# Arbitrary threshold, can be adjusted
|
||||
USAGE_THRESHOLD=90
|
||||
|
||||
@@ -18,7 +18,7 @@ while true; do
|
||||
# Check if usage is above threshold
|
||||
if [[ $USAGE -gt $USAGE_THRESHOLD ]]; then
|
||||
# Find the modification time of oldest file across all subdirectories
|
||||
OLDEST_MTIME=$(find "$TARGET_DIR" -type f -path "$EXCLUDE_DIR" -prune -o -type f -exec stat -c %Y {} \; | sort -n | head -1)
|
||||
OLDEST_MTIME=$(find "$TARGET_DIR" -type f ! -path "$EXCLUDE_DIR" -exec stat -c %Y {} \; | sort -n | head -1)
|
||||
|
||||
if [[ -z "$OLDEST_MTIME" ]]; then
|
||||
echo "No files found to delete"
|
||||
@@ -31,10 +31,11 @@ while true; do
|
||||
echo "Oldest file: $OLDEST_FILE (Age: $AGE_DAYS days)"
|
||||
|
||||
# Delete all files with the same age in all subdirectories using mtime
|
||||
find "${SUBDIRS[@]/#/$TARGET_DIR/}" -type f -mtime "$AGE_DAYS" -delete
|
||||
find "$TARGET_DIR" -type f ! -path "$EXCLUDE_DIR" -mtime "$AGE_DAYS" -delete
|
||||
echo "Deleted files with age $AGE_DAYS days"
|
||||
else
|
||||
echo "Disk usage is below threshold ($USAGE% < $USAGE_THRESHOLD%)"
|
||||
break
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
@@ -10,5 +10,5 @@ server {
|
||||
ssl_client_certificate /etc/nginx/certs/cloudflare-client.crt;
|
||||
ssl_verify_client on;
|
||||
|
||||
return 301 https://builds.vcmi.eu$request_uri;
|
||||
return 301 https://download.vcmi.eu$request_uri;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user