3697: Include sensible error messages for LMTP protocol (backport #3696) r=mergify[bot] a=mergify[bot]
Running into the rate limit yields difficult to debug log messages by the smtp container. Specifically the `Temporary user lookup failure` message by the smtp container is misleading.
## Example
Although this is running on Podman, the bugs are in the Python code and almost certainly are not influenced by the host infrastructure. (Leaving aside that I likely have a configuration problem, because the client IP address is not passed along correctly. But the present fix applies nevertheless and is not related to any specific cause of the rate limit triggering.)
### smtp logs
```shell
> podman logs --since "2024-12-25T07:33:31" --until "2024-12-25T07:33:33" systemd-mail-smtp
Dec 25 08:33:31 example postfix/smtpd[398]: connect from front[10.115.0.96]
INFO:root:Connect
Dec 25 08:33:31 example postfix/smtpd[398]: 6774324DE71C1: client=systemd-mail-front[10.115.0.96]
INFO:root:Connect
Dec 25 08:33:31 example postfix/cleanup[428]: 6774324DE71C1: message-id=<CAPhkJv+GTxVtwn6eNbBzPscohn6fgkhrYd2gEpUm2prr-5_7bg@mail.gmail.com>
Dec 25 08:33:32 example postfix/qmgr[376]: 6774324DE71C1: from=<SRS0=O1up=TS=gmail.com=fabiamos@example.com>, size=3968, nrcpt=1 (queue active)
Dec 25 08:33:32 example postfix/lmtp[429]: 6774324DE71C1: host front[10.115.0.96] said: 451 4.3.0 <fabian@example.com> Temporary user lookup failure (in reply to RCPT TO command)
Dec 25 08:33:32 example postfix/lmtp[429]: connect to front[10.115.0.9]:2525: Connection refused
Dec 25 08:33:32 example postfix/lmtp[429]: 6774324DE71C1: to=<fabian@example.com>, orig_to=<me+fancy@example.com>, relay=none, delay=0.63, delays=0.61/0.01/0.01/0, dsn=4.4.1, status=deferred (connect to front[10.115.0.9]:2525: Connection refused)
```
### admin logs
```shell
> podman logs --since "2024-12-25T07:33:31" --until "2024-12-25T07:33:33" systemd-mail-admin
10.115.0.96 - - [25/Dec/2024:08:33:31 +0100] "GET /internal/auth/email HTTP/1.0" 200 0 "-" "-"
[2024-12-25 08:33:32,030] WARNING in limiter: Authentication attempt from 10.115.0.99 has been rate-limited.
[2024-12-25 08:33:32,030] ERROR in app: Exception on /internal/auth/email [GET]
Traceback (most recent call last):
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/venv/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/mailu/internal/views/auth.py", line 27, in nginx_authentication
status, code = nginx.get_status(flask.request.headers['Auth-Protocol'], 'ratelimit')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/mailu/internal/nginx.py", line 140, in get_status
return status, codes[protocol]
~~~~~^^^^^^^^^^
KeyError: 'lmtp'
10.115.0.96 - - [25/Dec/2024:08:33:32 +0100] "GET /internal/auth/email HTTP/1.0" 200 0 "-" "-"
```
## What type of PR?
bug-fix
## What does this PR do?
### Related issue(s)
None
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly - not an enhancement
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file. - is a minor change
<hr>This is an automatic backport of pull request #3696 done by [Mergify](https://mergify.com).
Co-authored-by: Fabian Stanke <me+github@fmos.at>
These are difficult to debug log messages, when instead of an error message one gets a stack trace with `KeyError: 'lmtp'`
(cherry picked from commit ec1e49d137)
3692: Ensure mobileconfig has the right content-type (backport #3691) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Ensure Apple mobileconfig is served using the right Content-Type
### Related issue(s)
- #3684
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3691 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3653: Don't check empty passwords against HIBP (backport #3650) r=nextgens a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Don't check empty passwords against HIBP; Apparently some password managers will trigger a race condition otherwise
### Related issue(s)
- closes#3633
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3650 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3649: Upgrade snappymail to v2.38.2 (backport #3648) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Upgrade snappymail to v2.38.2. This is a security fix for [GHSA-2rq7-79vp-ffxm](https://github.com/the-djmaze/snappymail/security/advisories/GHSA-2rq7-79vp-ffxm) (mXSS)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3648 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3623: alpine 3.20.3 (backport #3622) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Upgrade from alpine 3.20.0 to alpine 3.20.3 ; We need a fix for [CVE-2024-5535](https://security.alpinelinux.org/vuln/CVE-2024-5535)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3622 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3619: Filter logs line based and in binary mode without decoding utf-8 (backport #3618) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
try at fixing decoding errors when filtering logs
### Related issue(s)
- closes#3398
<hr>This is an automatic backport of pull request #3618 done by [Mergify](https://mergify.com).
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
3616: Ensure healthchecks timeout (backport #3608) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Ensure healthchecks timeout
### Related issue(s)
- close#3398
- close#3602
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3608 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3614: Fix http2 (backport #3613) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Fix http2; it does not require IPv6 nor certificates
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3613 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3559: Fix#3531 (backport #3557) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Ensure we have both RSA and ECDSA certs when using letsencrypt now that the default behaviour from certbot has changed.
This is only important for new installs, not those renewing existing certs.
### Related issue(s)
- closes#3531
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3557 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3480: Maybe fix#3402 (backport #3465) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Fix a potential problem with SO_REUSEADDR that may prevent admin from starting up
### Related issue(s)
- close#3402
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3465 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3469: Fix overrides (backport #3468) r=nextgens a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Fix broken overrides introduced in 2024.06.17
### Related issue(s)
- closes#3467
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3468 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3464: Fix#3450 (backport #3463) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Ensure we can do more than 100 parallel sessions.
This will use more RAM.
### Related issue(s)
- close#3450
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3463 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
3439: fix container name r=mergify[bot] a=Pegoku
## What type of PR?
documentation
## What does this PR do?
fixed the container name.
### Related issue(s)
none
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Pegoku <81113533+Pegoku@users.noreply.github.com>
3424: Fix#3411 (backport #3423) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Ensure we don't nuke all web-sessions when a password is changed.
This was reported with the command line but I suspect the API is affected too.
### Related issue(s)
- close#3411
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3423 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
3422: Fix 3420 - fixed syntax errors in certwatcher.py (backport #3421) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Fixes syntax errors in certwatcher.py that resulted in dovecot not being restarted upon detection of changed certificate files.
### Related issue(s)
- Auto close an issue like: closes#3420
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3421 done by [Mergify](https://mergify.com).
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
3412: misc fixes (backport #3407) r=mergify[bot] a=mergify[bot]
## What type of PR?
bug-fix
## What does this PR do?
Update the documentation: clarify that dovecot also needs to be reloaded if custom certs are in use
Fix a 'fatal error' that may be encountered when using snappymail
Fix ``INBOUND_TLS_ENFORCE`` (something you should never use)
Fix ``DEFAULT_QUOTA``
Increase the size of php/nginx buffers on webmail
Maybe fix utf-8 decoding problems in socrate
Maybe fix utf-8 problems in fetchmail folder names
### Related issue(s)
- closes#3401
- closes#3405
- closes#3403
- closes#3379
- closes#3272
- closes#2996
- #3398
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
<hr>This is an automatic backport of pull request #3407 done by [Mergify](https://mergify.com).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
Exception in thread Thread-2 (forward_text_lines):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run() File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/app/venv/lib/python3.12/site-packages/socrate/system.py", line 155, in forward_text_lines
current_line = src.readline()
^^^^^^^^^^^^^^
File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 166: invalid start byte
This was reported in #3398
(cherry picked from commit 61812ac32a)
