matrix-docker-ansible-deploy/YEAR-IN-REVIEW.md

2023

2023 was a year filled with many changes for matrix-docker-ansible-deploy. In this post, we're looking backward at some of the major changes that happened this year, as well as taking a glimpse of what's ahead in 2024.

2023 is probably the year of AI, with millions of people jumping aboard OpenAI's ChatGPT train. matrix-docker-ansible-deploy is no stranger to this and 2023 began with a PR from bertybuttface who added support for matrix-chatgpt-bot (see the changelog entry). While OpenAI's chat GPT website was frequently overloaded in the past, their API was up which made using this bot both convenient and more reliable.

AI aside, with the playbook's focus being containers, we're doubling down on being "container native" and becoming more interoperable for people hosting other containers on the Matrix server. In 2022, we've announced a few sibling Ansible playbooks, their use of Traefik and the possiblity of matrix-docker-ansible-deploy also switching to this reverse-proxy. This prediction materialized quickly. The largest change in the playbook in 2023 happened way back in February - matrix-docker-ansible-deploy starting the switch from nginx to Traefik and then quickly making Treafik the default reverse-proxy. As noted in the changelog entries, we envisioned a quick and complete elimination of matrix-nginx-proxy, but at the end of 2023, it hasn't happened yet. The playbook is already using Traefik as the front-most reverse-proxy, but nginx (via matrix-nginx-proxy) is still around - it has taken a step back and is only used internally for new setups. Work got to a stall due to:

• complexity: untangling the overly large and messy matrix-nginx-proxy component is difficult
• the current setup became "good enough" because nginx has become an internal implementation detail for those who have migrated to Traefik. Traefik is already the default public reverse-proxy and gives better possibilities to people wishing to run other web-exposed containers on their Matrix server via Docker Compose, other Ansible playbooks like mash-playbook (more about this one, below) or any other way.

matrix-nginx-proxy is no longer in the way of us being interoperable, but its ugly internal details are still there. It is one more proxy in the long chain of reverse-proxies we have and we'd like to cut it out. This would both make things simpler and also boost performance.

The delay in eliminating matrix-nginx-proxy has probably been welcome by many existing users who decided to postpone the Traefik migration a bit longer. In 2024, work on eliminating matrix-nginx-proxy will continue with rapid pace. People who are still using matrix-nginx-proxy as their front-most reverse-proxy will need to rework their setup. About a year of putting it off has been long enough.

This large Traefik reverse-proxy change was also accompanied by another internal change which began in 2022, but continued in 2023 - moving non-Matrix-related roles from being internal to the playbook to living their own life outside of it. Various roles were made more decoupled and moved outside of the playbook, so that other projects (like the mash-playbook Ansible playbook or other Ansible playbooks) could benefit from them. This led to the death of a few sibling playbooks (gitea-docker-ansible-deploy, nextcloud-docker-ansible-deploy, peertube-docker-ansible-deploy, vaultwarden-docker-ansible-deploy), but brought life to something better, which supports all these services and more.

mash-playbook is a new Ansible playbook that a few of us (matrix-docker-ansible-deploy contributors) have launched in 2023. It has quickly grown to supports 60+ services and aims to do the same for FOSS service hosting, as matrix-docker-ansible-deploy has done for Matrix - providing a clean and secure way to run a bunch of services in containers on a regular server (that is to say, without Kubernetes, etc.). Thanks to Traefik and Ansible role reuse, it's easy to host both mash-playbook services and matrix-docker-ansible-deploy services on the same server - see mash-playbook's interoperability documentation page. If you've been looking for a holiday project or your New Year's Resolutions list contains "self-hosting more services", then you're welcome to give this new playbook a try and join its Matrix room (#mash-playbook:devture.com).

Because many of the roles are now external to this playbook (defined in the requirements.yml file), running make roles (or better yet just roles via the just tool) becomes a necessity each time one pulls playbook updates (git pull). Pulling external roles happens via the ansible-galaxy command-line tool, but if available, the playbook would also use the much faster agru tool (developed by Aine from etke.cc this year).

With the internal (but important) details out of the way, we can now talk more about new features that landed in matrix-docker-ansible-deploy in 2023.

The following new bridges were added to the playbook in 2023:

• (2023-01-11) mautrix-slack, thanks to a PR by Cody Neiman (see the changelog entry)
• (2023-07-21) mautrix-gmessages, thanks to a PR by Shreyas Ajjarapu (see the changelog entry)
• (2023-08-23) mautrix-wsproxy for Apple iMessage bridging (when combined with the mautrix-imessage bridge running on your Mac or Android phone), thanks to a PR by Johan Swetzén

This brings the total number of bridges that the playbook supports up to 30. There are alternative bridge implementations for various networks and protocols, so the number of "unique bridged networks" is surely much smaller.

A few other major components and changes landed in 2023:

• (2023-02-10) The Draupnir moderation tool (successor to Mjolnir), thanks to a PR by FSG-Cat (see the changelog entry)
• (2023-02-10) Matrix User Verification Service to add Matrix Authentication Support to our Jitsi setup, thanks to a PR by Jakob S. from zakk gGmbH (see the changelog entry)
• (2023-02-25) The rageshake bug report server, thanks to a PR by Benjamin Kampmann (see the changelog entry)
• (2023-03-07) Sliding Sync proxy (currently a necessary component for Element X to work), thanks to: Benjamin Kampmann and FSG-Cat (see the changelog entry)
• (2023-03-12) synapse-auto-compressor to periodically and automatically run rust-synapse-compress-state, thanks to a PR by Aine from etke.cc (see the changelog entry)
• (2023-07-17) matrix-media-repo,  thanks to a PR by Michael Hollister from FUTO, the creators of the Circles app (see the changelog entry)
• (2023-08-31) SchildiChat Web client app (fork of Element Web), thanks to a PR by Aine from etke.cc (see the changelog entry)
• (2023-10-18) Postgres parameters auto-tuning, thanks to a PR by Aine from etke.cc (see the changelog entry)
• (2023-10-23) Enabling federation of the room directory for Synapse (see the changelog entry)

The most recent change in the list above (Enabling federation of the room directory for Synapse) has been somewhat controversial as it goes against upstream defaults for Synapse. Nevertheless, we believe it promotes the well-being of the Matrix Federation by improving room discovery.

Matrix Federation Stats (containing the percentage of servers publishing their room directory publicly) are posted to TWIM each week by Aine from etke.cc. The number of servers which currently published their room directory publicly stands at 26.6%, which is:

• 2.4% more than when it was when first published to TWIM (1 month earlier, in November)
• likely about 15+% more than from before we flipped the switch (in October)

Hopefully, Synapse defaults would also change the same way and we'd see the number of servers publicly listing their room directory grow faster.

With this configuration change in place, projects like MatrixRooms.info (made by etke.cc) and potentially others in the future, can discover, index the metadata (room address, title, topic, number of users, etc.) and make public rooms browsable & searchable across the whole Matrix Federation. It'd be great if users joining Matrix could more easily find interesting communities that match their interests!

On the media side of things, besides Jitsi getting better Matrix integration (via the aforementioned Matrix User Verification Service), we've also had some Coturn security tightening as well as performance optimizations for configurations exposing lots of network ports.

Element Call seems to have become a nice and polished product lately (as proclaimed in The Matrix Holiday Update 2023), so 2024 is likely the year we'll see support for it in the playbook. Element Call depends on the LiveKit streaming server (which is also useful to developers even by itself), so the first step is likely to see LiveKit support in mash-playbook via a reusable Ansible role. Such a LiveKit Ansible role could later easily land in matrix-docker-ansible-deploy and an Element Call static website could be hooked to it.

Besides these highlights, there were many other relatively large changes announced in our CHANGELOG and hundreds of other more minor (but still important) playbook changes that didn't get a mention.

We have hundreds of contributors to thank for their hard work on making Matrix self-hosting better for all of us! It should be noted that support comes in many shapes, not only in raw code commits and financial help (via donations or using the etke.cc managed Matrix hosting service which is based on matrix-docker-ansible-deploy). It also comes in the shape of code reviews, helping others with issues, reporting new issues, participating in our support room on Matrix (#matrix-docker-ansible-deploy:devture.com), etc. To everyone who has been there to make matrix-docker-ansible-deploy better in 2023, thank you! 🙇‍♂️

2022

For matrix-docker-ansible-deploy, 2022 started with breaking the Synapse monopoly by adding support for the Dendrite Matrix homeserver in early January. This required various internal changes so that the Ansible playbook would not be Synapse-centric anymore. This groundwork paved the way for continuing in this direction and we added support for Conduit in August.

When it comes to the matrix-docker-ansible-deploy Ansible playbook, 2022 was the year of the non-Synapse homeserver implementation. In practice, none of these homeserver implementations seem ready for prime-time yet and there is no migration path when coming from Synapse. Having done our job of adding support for these alternative homeserver implementations, we can say that we're not getting in the way of future progress. It's time for the Dendrite developers to push harder (development-wise) and for the Synapse developers to take a well-deserved long (infinite) break, and we may get to see more people migrating away from Synapse in the next year(s).

Support for the following new bridges was added:

• Postmoogle for bi-directional email bridging, which supersedes my old and simplistic email2matrix one-way bridge-bot
• mautrix-discord
• go-skype-bridge
• matrix-appservice-kakaotalk

Support for the following new bots was added:

• buscarron bot
• Honoroit bot
• matrix-registration-bot
• matrix-hookshot
• maubot

Support for the following new components and services was added:

• BorgBackup
• Cactus Comments
• Cinny client support
• ntfy notifications
• matrix-ldap-registration-proxy
• matrix_encryption_disabler support
• synapse-s3-storage-provider to stop the Synapse media store from being a scalability problem. This brought along another feature - an easier way to customize the Synapse container image without having to fork and self-build all of it from scratch

Besides these major user-visible changes, a lot of work also happened under the hood:

• we made major improvements to Synapse workers - adding support for stream writers and for running multiple workers of various kinds (federation senders, pushers, background task processing workers, etc.)
• we improved the compatibility of (Synapse + workers) with the rest of the playbook by introducing a new matrix-synapse-reverse-proxy-companion-service service
• we started splitting various Ansible roles out of the Matrix playbook and into independent roles (e.g. matrix-postgres -> ansible-role-postgres), which could be included in other Ansible playbooks. In fact, these roles already power a few interesting other sibling playbooks:
  • gitea-docker-ansible-deploy, for deploying a Gitea (self-hosted Git service) server
  • nextcloud-docker-ansible-deploy, for deploying a Nextcloud groupware server
  • vaultwarden-docker-ansible-deploy, for deploying a Vaultwarden password manager server (unofficial Bitwarden compatible server)

These sibling playbooks co-exist nicely with one another due to using Traefik for reverse-proxying, instead of trying to overtake the whole server by running their own nginx reverse-proxy. Hopefully soon, the Matrix playbook will follow suit and be powered by Traefik by default.

Last, but not least, to optimize our etke.cc managed Matrix hosting service's performance (but also individual Ansible playbook runs for people self-hosting by themselves using the playbook), we've improved playbook runtime 2-5x by employing various Ansible tricks.