3729: Allow setting collation via env variable and add uvloop r=mergify[bot] a=Grennith
## What type of PR?
Enhancement / Bugfix
## What does this PR do?
As of https://github.com/Mailu/Mailu/pull/3701, the collation shall be set by overwriting `SQLALCHEMY_DATABASE_URI` to contain the collation of the related DB. However, this is currently not possible in the Helm chart of Mailu at all. It's statically set there and would also require not setting DB_NAME etc. to not have it overwritten, see https://github.com/Mailu/Mailu/blob/master/core/admin/mailu/configuration.py#L144
Additionally, uvloop is added to the prod requirements of which postfix-mta-sts-resolver makes use of.
### Related issue(s)
- Mention an issue like: #3449
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ x ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Till Skrodzki <till@mueskro.de>
3758: added idna function to perform puny encoding on IDN domains r=mergify[bot] a=Jumper78
## What type of PR?
bug-fix
## What does this PR do?
### Related issue(s)
- fixes issue where DKIM signatures from domains with IDN are not accepted by some mail servers: closes#3743
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Jumper78 <52802286+Jumper78@users.noreply.github.com>
3755: Update Roundcube to 1.6.10 r=mergify[bot] a=ctrl-i
## What type of PR?
Update
## What does this PR do?
Updates roundcube to the latest version - 1.6.10
The new version of roundcube includes various fixes, is the next service release and considered stable.
The change log can be found [here](https://github.com/roundcube/roundcubemail/releases/)
### Related issue(s)
- None
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: ctrl-i <1422608+ctrl-i@users.noreply.github.com>
3742: Fixed "core" Docker image to allow x86_32 building, updated documenta… r=mergify[bot] a=vparres
## What type of PR?
bugfix / documentation
## What does this PR do?
Allow building a i686 linux image on a amd64 machine by switching from `uname -m` architecture detection to `apk --print-arch` to better reflect the binary distribution architecture instead of the current Kernel architecture and adding the rust toolchain in dependencies.
Also updated the documentation to mention SSE4.2 requirement and reference to the mailu issue i opened earlier.
I didn't wanted to add any kind of automated building changes in this PR before discussions, as it may add unneeded pressure on mailu pipelines. To solve the issue with rspamd mentioned in #3713, only the `base` and the `antispam` images needs to be rebuilt on i686, and it can be done locally if needed.
### Related issue(s)
- closes#3713
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Vincent PARRES-GACON <contact@vparres.me>
3748: Bump CREDENTIAL_ROUNDS to 13 r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Bump CREDENTIAL_ROUNDS to 13. As discussed on #mailu-dev, CPUs get faster.
Benchmark it using:
```
python3 -m timeit -n 1 -s "from passlib.hash import bcrypt_sha256" "bcrypt_sha256.using(rounds=13).hash('password')"
```
### Related issue(s)
- #1753
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3749: Ensure tests actually work r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Ensure tests in CI actually work
### Related issue(s)
- #3587
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- "core" Docker image : Switched from uname -m architecture detection to apk --print-arch to better reflect the binary distribution architecture instead of the current Kernel architecture. This allows building a i686 linux image on a amd64 machine.
- "core" Docker image : Adding rust toolchain in dependancies to allow python modules to build properly on uncommonly supported architectures (like i686)
- Documentation : Updated setup requirement to mention SSE4.2 requirement
- Documentation : Updated faq with the full explanation for the SSE4.2 requirement and the reference to the mailu issue.
3739: Fix the webpack build due to dependOn issue r=mergify[bot] a=kaiyou
## What type of PR?
Bug-fix
## What does this PR do?
As stated in Webpack documentation, when using multiple entrypoints and dependencies, it is recommended to export the runtime as a single separated chunk.
See: https://webpack.js.org/guides/code-splitting/#entry-dependencies
### Related issue(s)
- closes#3738
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file. -> this is a minor build change
Co-authored-by: kaiyou <dev@kaiyou.fr>
My previous fix attempt only made it clear that the issue was
in the runtime and an upstream issue with Webpack, but did not
really fix things.
Since Webpack 5.96.0, especially since
420d0d0eed
Webpack does not generate JS for asset chunks, which breaks having
a single entry with both JS and asset chunks.
The logo can easily be moved to a separate entry.
3735: Fix `clamav` path to allow for updates r=mergify[bot] a=nazar-pc
## What type of PR?
Bug-fix, documentation
## What does this PR do?
### Related issue(s)
Closes https://github.com/Mailu/Mailu/issues/3673 by placing ClamAV files under `mailu/clamav` instead of `mailu/filter/clamav`.
Users will want to change their `docker-compose.yml` accordingly and remove `mailu/filter/clamav` after upgrade.
I also updated ClamAV version while I was at it (I didn't find any breaking changes in the changelog), though [the latest release is not pushed to this image yet](https://github.com/Cisco-Talos/clamav/issues/1442). Also I'm wondering why is it using exact version instead of `:1` or `:1.4` for example, but decided to not change that to make it less controversial.
## Prerequisites
This will not affect existing setups, though it would be nice to notify users somehow.
<!-- Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file. -->
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
With https://github.com/Mailu/Mailu/pull/3701 the remark was left to set the DB collation for MariaDB setups. However, the Helm chart has no option to overwrite the SQLAlchemy URI really. It selfs DB_USER, DB_PW, DB_NAME and DB_HOST and thus triggers the Alchemy URI to be overwritten (and it overwrites it statically as well...).
This commit adds the parameter / environment variable DB_APPENDIX allowing for, e.g., '?collation=utf8mb4_unicode_ci' to be set using an environment variable which the Helm chart can then set.
3722: Ensure we always use Mailu for sending emails in thunderbird r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Set useGlobalPreferredServer=false in autoconfig to ensure we always use Mailu's SMTP if there is more than one account configured.
The previous behaviour made no sense; it was set that way because the template at https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat makes it the default.
### Related issue(s)
- close#3721
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3709: Clarify ip listen address setting r=mergify[bot] a=nextgens
## What type of PR?
documentation
## What does this PR do?
Clarify ip listen address setting in setup to avoid open-relays.
Thanks to `@Cenness` for reporting it and suggesting a better wording.
### Related issue(s)
- closes#3680
- closes#3683
- #3690
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3440: Update reverse proxy documentation for using Traefik on a different host r=mergify[bot] a=Diman0
## What type of PR?
documentation
## What does this PR do?
It adds an extra section to the reverse proxy documentation. It provides an example on how to use Traefik on a different host than the host running Mailu. Now we will have documented both use cases where the reverse proxy is on the same host or a different host than Mailu.
### Related issue(s)
n/a
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
3699: Add the mariadb connector as per 3449 r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Add the mariadb connector as per #3449.
MariaDB has no support for utf8mb4_0900_ai_ci which is the new default since MySQL version 8.0. In the current sqlalchemy version shipped with mailu, the mysqlconnector sets utf8mb4_0900_ai_ci as the collation to use when connecting. This causes all MariaDB connections to fail.
To fix the issue, either use the right connector or ensure it's configured with the right collation:
```
SQLALCHEMY_DATABASE_URI=mysql+mysqlconnector://<user>:<passwd>`@<host>/<database>?collation=utf8mb4_unicode_ci`
```
### Related issue(s)
- closes#3449
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3696: Include sensible error messages for LMTP protocol r=mergify[bot] a=fmos
Running into the rate limit yields difficult to debug log messages by the smtp container. Specifically the `Temporary user lookup failure` message by the smtp container is misleading.
## Example
Although this is running on Podman, the bugs are in the Python code and almost certainly are not influenced by the host infrastructure. (Leaving aside that I likely have a configuration problem, because the client IP address is not passed along correctly. But the present fix applies nevertheless and is not related to any specific cause of the rate limit triggering.)
### smtp logs
```shell
> podman logs --since "2024-12-25T07:33:31" --until "2024-12-25T07:33:33" systemd-mail-smtp
Dec 25 08:33:31 example postfix/smtpd[398]: connect from front[10.115.0.96]
INFO:root:Connect
Dec 25 08:33:31 example postfix/smtpd[398]: 6774324DE71C1: client=systemd-mail-front[10.115.0.96]
INFO:root:Connect
Dec 25 08:33:31 example postfix/cleanup[428]: 6774324DE71C1: message-id=<CAPhkJv+GTxVtwn6eNbBzPscohn6fgkhrYd2gEpUm2prr-5_7bg@mail.gmail.com>
Dec 25 08:33:32 example postfix/qmgr[376]: 6774324DE71C1: from=<SRS0=O1up=TS=gmail.com=fabiamos@example.com>, size=3968, nrcpt=1 (queue active)
Dec 25 08:33:32 example postfix/lmtp[429]: 6774324DE71C1: host front[10.115.0.96] said: 451 4.3.0 <fabian@example.com> Temporary user lookup failure (in reply to RCPT TO command)
Dec 25 08:33:32 example postfix/lmtp[429]: connect to front[10.115.0.9]:2525: Connection refused
Dec 25 08:33:32 example postfix/lmtp[429]: 6774324DE71C1: to=<fabian@example.com>, orig_to=<me+fancy@example.com>, relay=none, delay=0.63, delays=0.61/0.01/0.01/0, dsn=4.4.1, status=deferred (connect to front[10.115.0.9]:2525: Connection refused)
```
### admin logs
```shell
> podman logs --since "2024-12-25T07:33:31" --until "2024-12-25T07:33:33" systemd-mail-admin
10.115.0.96 - - [25/Dec/2024:08:33:31 +0100] "GET /internal/auth/email HTTP/1.0" 200 0 "-" "-"
[2024-12-25 08:33:32,030] WARNING in limiter: Authentication attempt from 10.115.0.99 has been rate-limited.
[2024-12-25 08:33:32,030] ERROR in app: Exception on /internal/auth/email [GET]
Traceback (most recent call last):
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/mailu/internal/views/auth.py", line 27, in nginx_authentication
status, code = nginx.get_status(flask.request.headers['Auth-Protocol'], 'ratelimit')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/mailu/internal/nginx.py", line 140, in get_status
return status, codes[protocol]
~~~~~^^^^^^^^^^
KeyError: 'lmtp'
10.115.0.96 - - [25/Dec/2024:08:33:32 +0100] "GET /internal/auth/email HTTP/1.0" 200 0 "-" "-"
```
## What type of PR?
bug-fix
## What does this PR do?
### Related issue(s)
None
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly - not an enhancement
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file. - is a minor change
Co-authored-by: Fabian Stanke <me+github@fmos.at>
3691: Ensure mobileconfig has the right content-type r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Ensure Apple mobileconfig is served using the right Content-Type
### Related issue(s)
- #3684
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3678: alpine3.21.0 r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Upgrade to alpine-3.21.0
### Related issue(s)
- #3677
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3650: Don't check empty passwords against HIBP r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Don't check empty passwords against HIBP; Apparently some password managers will trigger a race condition otherwise
### Related issue(s)
- closes#3633
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3647: upgrade to snuffleupagus v0.11 r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
- upgrade snuffleupagus to v0.11
- filter php:// wrapper types and only allow those we currently require (hardening)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3648: Upgrade snappymail to v2.38.2 r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Upgrade snappymail to v2.38.2. This is a security fix for [GHSA-2rq7-79vp-ffxm](https://github.com/the-djmaze/snappymail/security/advisories/GHSA-2rq7-79vp-ffxm) (mXSS)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
