* move log/filter functions into own file
* add ShellCheck global directives
* use new function for tracking logs
The new function, called `_send_email_with_mid`, aligns with suggestions
from @polarethene and is heavily simplified compared to its predecessor
`_send_email_and_get_id`. New helpers will be introduced to filter logs
according to the MID constructed in this function.
* new filters for searching logs with MID
* use new filters (and sending) functions
* add new helper for asserting non-existence of log message
* use new filters in tests
* Apply suggestions from code review
- `_mid` / `MID` => `_msgid` / `MSG_ID`
- Revised documentation / tooltip comments
* Apply suggestions from code review
* fix tests
* use more distinct names for MSG_ID headers
* update `_filter_service_log` to not use `-i -E`
Moreover, I added a function to print the whole mail log. Appropriate
comments were added to this function to indicate that one should only
use this function when necessary.
* adjust helpers to new helper filter
* follow-up of previous commit
* add CHANGELOG entry
* Apply suggestions from code review
* chore: Update OAuth2 to use new log helper
* Apply suggestions from code review
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* added explicit `_regexp` filters for logs
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
- The old Dovecot wiki link needed to be updated to the new location.
- The new docs are not entirely compatible AFAIK, thus making the existing examples/docs a bit outdated / incompatible. A warning admonition has been added early on to raise awareness to the reader.
- Minor formatting revisions to the content.
* fix: Dovecot PassDB should restrict allowed auth mechanisms
This prevents PassDBs incompatible with certain auth mechanisms from logging failures which accidentally triggers Fail2Ban.
Instead only allow the PassDB to be authenticated against when it's compatible with the auth mechanism used.
* tests: Use `curl` for OAuth2 login test-cases instead of netcat
`curl` provides this capability for both IMAP and SMTP authentication with a bearer token. It supports both `XOAUTH2` and `OAUTHBEARER` mechanisms, as these updated test-cases demonstrate.
* chore: Add entry to `CHANGELOG.md`
The UID / GID shifted during a new release. Until DKIM handling is refactored in a new major release, this fix ensures the content maintains the expected `_rspamd` ownership.
* tests: OAuth2 - Replace Python `/userinfo` endpoint with Caddy
Better documented, easier flow and separation of concerns via Caddy.
The python code had additional noise related to setting up a basic API which is abstracted away via `Caddyfile` config that's dedicated to this task.
* tests: OAuth2 - Minimize noise + Improve test assertion
Caddyfile can use an Access Token instead of a JWT. Much smaller and correct for this OAuth2 configuration. This new value has been documented inline.
Likewise the `sub` field returned is not important to this test. `email_verified` is kept as it may be helpful for further coverage testing.
The actual test-case has better assertions for success and failure by checking for Dovecot logs we expect instead of netcat response.
`oauth2` to `auth` for the Caddy container hostname is not necessary, just a more generic subdomain choice.
* tests: OAuth2 - Caddyfile `imap/xoauth2` route dynamic via query string
This way is more flexible and doesn't require modifying the `Caddyfile` directly, while still easy to use.
Additionally simplifies understanding the Caddyfile to maintainers by removing the `route` directive that was required to ensure a deterministic order of vars.
* tests: OAuth2 - `/imap/xoauth2` respond with IMAP commands for netcat
Since this is the only intended usage, might as well have it respond with the full file content.
* tests: OAuth2 - Implement coverage for `OAUTHBEARER`
Caddyfile route for `/imap/` now accepts any subpath to support handling both `xoauth2` and `oauthbearer` subpaths.
Both SASL mechanisms represent the same information, with `XOAUTH2` being a common mechanism to encounter defined by Google, whilst `OAUTHBEARER` is the newer variant standardized by RFC 7628 but not yet as widely adopted.
The request to `/userinfo` endpoint will be the same, only the `credentials` value to be encoded differs.
Instead of repeating the block for a similar route, this difference is handled via the Caddyfile `map` directive.
We match the path context (_`/xoauth2` or `/oauthbearer`, the `/imap` prefix was stripped by `handle_path` earlier_), when there is a valid match, `sasl_mechanism` and `credentials` map vars are created and assigned to be referenced by the later `respond` directive.
---
Repeat the same test-case logic, DRY with log asserts extracted to a common function call. This should be fine as the auth method will be sufficient to match against or a common failure caught.
* tests: OAuth2 - Minor revisions
Separate test cases and additional comment on creating the same base64 encoded credentials via CLI as an alternative to running Caddy.
Added a simple `compose.yaml` for troubleshooting or running the container for the `/imap/xoauth2` / `/imap/oauthbearer` endpoints.
* tests: OAuth2 - Route endpoints in Caddyfile with snippets instead
`reverse_proxy` was a bit more convenient, but the additional internal ports weren't really relevant. It also added noise to logging when troubleshooting.
The `import` directive with Snippet blocks instead is a bit cleaner, but when used in a single file snippets must be defined prior to referencing them with the `import` directive.
---
`compose.yaml` inlines the examples, with slight modification to `localhost:80`, since the Caddyfile examples `auth.example.test` is more relevant to the tests which can use it, and not applicable to troubleshooting locally outside of tests.
* chore: Add entry to `CHANGELOG.md`
* chore: Additional context on access token
I figured this was a useful comment to reference related to the setting if it's ever being changed or needs to be better understood (linked issue is a common failure that can be encountered related to this restriction).
This is a more explicit reminder for any future contributors that get thrown off by the usage of `sed` here and may be inclined to change it.
Add a link to reference a comment where it's already been explored what the alternative `sed` invocations available are.
* correct misc typos
We also seem to be favoring `behavior` over `behaviour`.
* bump MkDocs version
* resolve errors shown when buildg docs
* improve the Rspamd page
* behaviour -> behavior
Streamline the usage of this word. The majority used behavior, so I
opted to go with this way of spelling it.
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
docs: Rspamd DKIM config (`dkim_signing.conf`) example has been simplified via `path` + `selector` settings.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
In Docker Compose `.env` files are parsed properly when values are wrapped with quotes. Trailing white-space is also discarded, like it would be with shell variables.
This is not the case with `docker run` or other CRI like `podman` (_including it's compose equivalent support_). Those will parse the quotes to be included in a literal string value. Trailing white-space is also retained.
Hence a default with a trailing space is not compatible across CRI. This change documents the default with additional context on how to include a trailing white-space with a custom value for the users CRI choice. It additionally clearly communicates the opt-out value for this feature.
* feat: add support for MTA-STS for outgoing mails
* Hook-up mta-sts-daemon into basic process handling test
* fix: Call python script directly
The python3 shebang will run it, which will now meet the expectations of the process testing via pgrep. fail2ban has the same approach.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* ci: `.gitattributes` - Ensure `eol=lf` for shell scripts
- These files should always use LF for line endings during a checkout.
- `Dockerfile` does not like building with HereDoc `RUN` scripts that expect LF.
* chore: Disable `smtputf8` support in config
This was always configured disabled at runtime, better to just set explicitly in `main.cf` unless config diverges when Dovecot is enabled to opt-out of this feature.
* fix: `supervisor-app.conf` - Correct `postgrey` log location
Looks like this should have been like every other service and reference a log file(s) based on program name in the supervisor log directory.
* tests: Adjust log location for `postgrey_enabled.bats`
* docs: Debugging - Delivery failure from service downtime
Services may be temporarily down, such as when restarted when certificates are updated due to the `check-for-changes.sh` service. This is another known source of intermittent delivery failures.
* ci: Allow lint workflow to be manually triggered
Without this a different event must occur to trigger the workflow, which is inconvenient for automated PRs.
The file is managed by the `contributors.yml` workflow, no need for linting to be triggered on PRs for that change.
This should ideally skip the required check status for the lint workflow which cannot trigger implicitly for automated PRs. If this doesn't work the change should be reverted.
