1
0
mirror of https://github.com/spantaleev/matrix-docker-ansible-deploy.git synced 2024-12-12 08:43:55 +02:00
Commit Graph

114 Commits

Author SHA1 Message Date
Marcel Partap
183adec3d8 Merge remote-tracking branch 'origin/master' into synapse-workers 2021-01-23 15:04:11 +01:00
Slavi Pantaleev
a56cb34850 Notify people if /matrix/postgres/data-auto-upgrade-backup exists 2021-01-23 14:14:45 +02:00
Slavi Pantaleev
a2422c458a Notify of remaining matrix-postgres local data in a better way 2021-01-23 14:04:51 +02:00
Slavi Pantaleev
f085362149 Fix some Postgres CLI scripts to target the correct database
Fixes a regression introduced in 95346f3117.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/814

Using `matrix_synapse_` variables in the `matrix-postgres` role is not
ideal, but.. this script belongs neither here, nor there.
We'll have it be like that for now.
2021-01-23 11:38:34 +02:00
Slavi Pantaleev
f9968b6981 Fix matrix_postgres_connection_password length check 2021-01-22 21:22:58 +02:00
Slavi Pantaleev
88addd71fc Fix Postgres imports going to the matrix DB by default
Well, they still do go to that DB by default,
but our docs give a better command to users, which would do the right
thing.
2021-01-22 17:39:08 +02:00
Slavi Pantaleev
bef0702fea Wait some more when starting Postgres during setup on ARM 2021-01-22 16:21:30 +02:00
Slavi Pantaleev
f9c1d62435 Fix Postgres database (-alpine) failing to start on ARM32 2021-01-22 13:52:55 +02:00
Slavi Pantaleev
95346f3117 Reorganize Postgres access (breaking change)
In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.

We also no longer have a default Postgres superuser password,
which improves security.

The changelog explains more as to why this was done
and how to proceed from here.
2021-01-22 13:26:12 +02:00
Slavi Pantaleev
1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev
05ca9357a8 Add .service suffix to systemd units list
We'll be adding `.timer` units later on, so it's good to be
more explicit.
2021-01-14 23:02:10 +02:00
Marcel Partap
cd8100544b Merge remote-tracking branch 'origin/master' into synapse-workers
Sync with upstream
2021-01-08 20:58:50 +01:00
Slavi Pantaleev
fb83eccf99 Relocate SQL template file 2021-01-03 00:58:31 +02:00
Slavi Pantaleev
86da489b9b Never fail when stopping systemd service during (SQLite -> Postgres) migration
We need to suppress systemd service-stopping requests in certain rare
cases like https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/771

That issue seems to describe a case, where a migration from mxisd to
ma1sd was happening (DB files had just been moved), and then we were
attemping to stop `matrix-ma1sd.service` so we could import that database into
Postgres. However, there's neither `matrix-mxisd.service`, nor
`matrix-ma1sd.service` after `migrate_mxisd.yml` had just run, so
stopping `matrix-ma1sd.service` was failing.
2020-12-29 10:31:20 +02:00
Slavi Pantaleev
21662af3be Archive database only after additional_psql_statements_list had executed 2020-12-23 14:12:11 +02:00
Stuart Mumford
019a4d7dcd Use role relative paths for things 2020-12-23 11:34:48 +00:00
Slavi Pantaleev
ad1425eee4 Add pgloader self-building support (for ARM) 2020-12-23 09:08:54 +02:00
Slavi Pantaleev
8675dedbdb Add support for automatic (nedb -> Postgres) migration to matrix-appservice-slack 2020-12-22 19:56:52 +02:00
Slavi Pantaleev
715bdf2c64 Add support for automatic (nedb -> Postgres) migration to mx-appservice-irc 2020-12-22 19:32:43 +02:00
Slavi Pantaleev
1bd5c240e5 Add support for executing additional DB migration statements
In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.
2020-12-15 23:18:29 +02:00
Slavi Pantaleev
dd797ba6a7 Fix Postgres database importing/upgrading conflicts
We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).
2020-12-14 22:28:20 +02:00
transcaffeine
13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev
af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev
c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev
dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev
f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
Slavi Pantaleev
cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev
c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev
6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev
183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00
Slavi Pantaleev
46a4034d3e Use "password" for additional Postgres databases, not "pass"
Being more explicit sounds better.
2020-12-14 00:43:03 +02:00
Slavi Pantaleev
3a037a5993 Ensure additional databases contain all the keys that we expect 2020-12-14 00:39:38 +02:00
Slavi Pantaleev
da4cb2f639 Do not use the postgresql_user/postgresql_db modules
While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.
2020-12-14 00:31:38 +02:00
Slavi Pantaleev
bbc09d013b Do not execute additional databases creation code if not necessary
The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.
2020-12-13 23:46:05 +02:00
Slavi Pantaleev
c765ceb270 Prevent weird loop error
> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).
2020-12-13 22:56:56 +02:00
Slavi Pantaleev
e2952f16f7 Determine matrix-postgres IP address without relying on jq
To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether
2020-12-13 22:45:48 +02:00
Slavi Pantaleev
f47e8a97e6 Make use of matrix_host_command_docker instead of hardcoding 2020-12-13 22:38:35 +02:00
Slavi Pantaleev
0641106370 Allow username of additional Postgres databases to be different
We'll most likely use one that matches the database name, but
it's better to have it configurable.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
527d5f57d5 Relocate Postgres additional database creation logic
Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
dac0d3a682 Add default matrix_postgres_additional_databases 2020-12-13 21:07:16 +02:00
Slavi Pantaleev
77a5c7cf3c Merge branch 'master' into postgres-per-default 2020-12-13 21:04:15 +02:00
Slavi Pantaleev
47613e5a27 Remove synapse-janitor support
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/746
2020-12-11 23:24:42 +02:00
transcaffeine
d9f4914e0d
WIP: postgres: create databases for all services
If a service is enabled, a database for it is created in postgres with a uniqque password. The service can then use this database for data storage instead of relying on sqlite.
2020-12-10 18:26:22 +01:00
Slavi Pantaleev
d08b27784f Fix systemd services autostart problem with Docker 20.10
The Docker 19.04 -> 20.10 upgrade contains the following change
in `/usr/lib/systemd/system/docker.service`:

```
-BindsTo=containerd.service
-After=network-online.target firewalld.service containerd.service
+After=network-online.target firewalld.service containerd.service multi-user.target
-Requires=docker.socket
+Requires=docker.socket containerd.service
Wants=network-online.target
```

The `multi-user.target` requirement in `After` seems to be in conflict
with our `WantedBy=multi-user.target` and `After=docker.service` /
`Requires=docker.service` definitions, causing the following error on
startup for all of our systemd services:

> Job matrix-synapse.service/start deleted to break ordering cycle starting with multi-user.target/start

A workaround which appears to work is to add `DefaultDependencies=no`
to all of our services.
2020-12-10 11:43:20 +02:00
Marcel Partap
b73ac965ac Merge remote-tracking branch 'origin/master' into synapse-workers 2020-12-01 21:24:26 +01:00
Slavi Pantaleev
796c752b60 Ensure Postgres passwords are not longer than 99 characters
Complements https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/732
2020-11-26 09:51:48 +02:00
Slavi Pantaleev
75f9fde7a4 Remove some more -v usage
Continuation of 1fca917ad1.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/722
2020-11-25 10:49:59 +02:00
Slavi Pantaleev
1fca917ad1 Replace some -v instances with --mount
`-v` magically creates the source destination as a directory,
if it doesn't exist already. We'd like to avoid this magic
and the potential breakage that it might cause.

We'd rather fail while Docker tries to find things to `--mount`
than have it automatically create directories and fail anyway,
while having contaminated the filesystem.

There's a lot more `-v` instances remaining to be fixed later on.
This is just some start.

Things like `matrix_synapse_container_additional_volumes` and
`matrix_nginx_proxy_container_additional_volumes` were not changed to
use `--mount`, as options for each one are passed differently
(`ro` is `ro`, but `rw` doesn't exist and `slave` is `bind-propagation=slave`).
To avoid breaking people's custom volume mounts, we keep it as it is for now.

A deficiency with `--mount` is that it lacks the `z` option (SELinux
ownership changes), and some of our `-v` instances use that. I'm not
sure how supported SELinux is for us right now, but it might be,
and breaking that would not be a good idea.
2020-11-24 10:26:05 +02:00