self-hosted/Mailu
1
0
Fork 0
mirror of https://github.com/Mailu/Mailu.git synced 2024-12-12 10:45:38 +02:00
Code Issues Releases Activity

Merge #2403

Browse Source 
2403: Feature: switch CI/CD from build to buildx r=mergify[bot] a=Diman0

## What type of PR?

Feature and enhancement

## What does this PR do?

Switch from docker build to buildx for CI/CD.
    - The main workflow file has been optimised and simplified.
    - Images are built in parallel when building locally resulting in much faster build times.
    - The github action workflow is about 50% faster.
    - Arm images are built as well. These images are not tested due to restrictions of github actions (no arm runners). The tags of the images have -arm appended to it. The arm images are built on merge on master and release branch (x.y). They do not influence the normal CI/CD workflow used for bors (for PR) and real releases (merge on master and branch x.y for x86_64). 
    - Arm images (and normal x86_64 images) can also be built locally.
    - Reusable workflow is introduced for building, testing and deploying the images. This allows the workflow to be reused for other purposes in the future.
    - Workflow can be manually triggered. This allows forked Mailu projects to also use the workflow for building images.

The main workflow makes use of github actions cache to store the cache layer. This layer is used to quickly rebuilt the images in the testing step and deploy step.

Unfortunately the building the arm images fails sometimes due to timeouts. Sometimes the connection to github actions cache is very slow. Restarting the workflow from the last failed step resolves this. I have not observed this with the normal build.

Just as previous time, you can use a forked project for testing the changes (https://github.com/Diman0/Mailu_Fork). You should still have owner access. I have created branch 1.11 for testing. You can see I already push 4 times to branch 1.11 (current version is 1.11.3).

### Related issue(s)
- Mention an issue like: #001
- closes #2383 
- closes #1830
- closes #1200

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
This commit is contained in:
bors[bot] 2022-08-19 13:19:13 +00:00 committed by GitHub
commit 53de7b7d60
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
25 changed files with 918 additions and 943 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

678
.github/workflows/CI.yml vendored
View File

@ -1,678 +0,0 @@
name: CI
on:
push:
branches:
- staging
- testing
- '1.8'
- '1.9'
- master
# test branches, e.g. test-debian
- test-*
concurrency: ci-${{ github.ref }}
###############################################
# REQUIRED secrets
# DOCKER_UN: ${{ secrets.Docker_Login }}
# Username of docker login for pushing the images to repo $DOCKER_ORG and $DOCKER_ORG_TESTS
# DOCKER_PW: ${{ secrets.Docker_Password }}
# Password of docker login for pushing the images to repo $DOCKER_ORG and $DOCKER_ORG_TESTS
# DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
# The docker repository where the images are pushed to.
# DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
# The docker repository for test images. Only used for the branch TESTING (BORS try).
# Add the above secrets to your github repo to determine where the images will be pushed.
################################################
# Code block that is used as one liner.
##!/bin/bash
#version=$( git tag --sort=version:refname --list "{{ env.MAILU_VERSION }}.*" | tail -1 )
#root_version=${version%.*}
#patch_version=${version##*.}
#if [ "$patch_version" == "" ]
#then
# pinned_version={{ env.MAILU_VERSION }}.0
#else
# pinned_version=$root_version.$(expr $patch_version + 1)
#fi
#echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
jobs:
build:
name: Build images
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Check docker-compose version
run: docker-compose -v
- name: Login docker
env:
DOCKER_UN: ${{ secrets.Docker_Login }}
DOCKER_PW: ${{ secrets.Docker_Password }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin
- name: Build all docker images
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
run: docker-compose -f tests/build.yml build
- name: Save all docker images
run: docker save ${{ env.DOCKER_ORG }}/admin ${{ env.DOCKER_ORG }}/clamav ${{ env.DOCKER_ORG }}/docs ${{ env.DOCKER_ORG }}/dovecot ${{ env.DOCKER_ORG }}/fetchmail ${{ env.DOCKER_ORG }}/nginx ${{ env.DOCKER_ORG }}/none ${{ env.DOCKER_ORG }}/postfix ${{ env.DOCKER_ORG }}/radicale ${{ env.DOCKER_ORG }}/snappymail ${{ env.DOCKER_ORG }}/roundcube ${{ env.DOCKER_ORG }}/rspamd ${{ env.DOCKER_ORG }}/setup ${{ env.DOCKER_ORG }}/traefik-certdumper ${{ env.DOCKER_ORG }}/unbound -o /images/images.tar.gz
test-core:
name: Perform core tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test core suite
run: python tests/compose/test.py core 2
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
test-fetchmail:
name: Perform fetchmail tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test fetch
run: python tests/compose/test.py fetchmail 2
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
test-filters:
name: Perform filter tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test clamvav
run: python tests/compose/test.py filters 3
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
test-snappymail:
name: Perform snappymail tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test snappymail
run: python tests/compose/test.py snappymail 2
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
test-roundcube:
name: Perform roundcube tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test roundcube
run: python tests/compose/test.py roundcube 2
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
test-webdav:
name: Perform webdav tests
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
run: docker load -i /images/images.tar.gz
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test webdav
run: python tests/compose/test.py webdav 2
env:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
deploy:
name: Deploy images
runs-on: ubuntu-latest
needs:
- build
- test-core
- test-fetchmail
- test-filters
- test-snappymail
- test-roundcube
- test-webdav
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to PR-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION for branch testing
if: ${{ env.BRANCH == 'testing' }}
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG_TESTS" >> $GITHUB_ENV
- name: Derive MAILU_VERSION for other branches than testing
if: ${{ env.BRANCH != 'testing' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging
if: ${{ env.BRANCH == 'staging' }}
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for master
if: ${{ env.BRANCH == 'master' }}
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
- name: Create folder for storing images
run: |
sudo mkdir -p /images
sudo chmod 777 /images
- name: Configure images folder for caching
# For staging we do not deploy images. So we do not have to load them from cache.
if: ${{ env.BRANCH != 'staging' }}
uses: actions/cache@v2
with:
path: /images
key: ${{ env.BRANCH }}-${{ github.run_id }}-${{ github.run_number }}
- name: Load docker images
if: ${{ env.BRANCH != 'staging' }}
run: docker load -i /images/images.tar.gz
- name: Deploy built docker images
env:
DOCKER_UN: ${{ secrets.Docker_Login }}
DOCKER_PW: ${{ secrets.Docker_Password }}
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
BRANCH: ${{ env.BRANCH }}
run: bash tests/deploy.sh
tag-release:
runs-on: ubuntu-latest
needs:
- deploy
steps:
- uses: actions/checkout@v2
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
- name: Derive MAILU_VERSION amd DOCKER_ORG
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master' }}
shell: bash
env:
DOCKER_ORG: ${{ secrets.DOCKER_ORG }}
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG=$DOCKER_ORG" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for normal release x.y
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master' }}
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
- name: Create tag for branch x.y.
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master' && env.PINNED_MAILU_VERSION != '' }}
shell: bash
run: |
echo git tag ${{ env.PINNED_MAILU_VERSION }} $(/usr/bin/git rev-parse HEAD)
git tag ${{ env.PINNED_MAILU_VERSION }} $(/usr/bin/git rev-parse HEAD)
git push origin ${{ env.PINNED_MAILU_VERSION }}
- name: Create release for tag x.y.z.
if: ${{ env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master' && env.PINNED_MAILU_VERSION != '' }}
uses: ncipollo/release-action@v1
with:
bodyFile: "RELEASE_TEMPLATE.md"
tag: ${{ env.PINNED_MAILU_VERSION }}
token: ${{ secrets.GITHUB_TOKEN }}
# This job is watched by bors. It only complets if building,testing and deploy worked.
ci-success:
name: CI-Done
#Returns true when none of the **previous** steps have failed or have been canceled.
if: ${{ success() }}
needs:
- deploy
runs-on: ubuntu-latest
steps:
- name: CI/CD succeeded.
run: exit 0

101
.github/workflows/arm.yml vendored Normal file
View File

@ -0,0 +1,101 @@
name: start-linux-arm
on:
push:
branches:
- '1.9'
- master
concurrency: ci-arm-${{ github.ref }}
###############################################
# REQUIRED secrets
# ${{ secrets.Docker_Login }}
# Username of docker login for pushing the images to repo env.DOCKER_ORG and env.DOCKER_ORG_TESTS
# ${{ secrets.Docker_Password }}
# Password of docker login for pushing the images to repo env.DOCKER_ORG and env.DOCKER_ORG_TESTS
# Add the above secrets to your github repo to determine where the images will be pushed.
################################################
# REQUIRED global variables
# DOCKER_ORG, docker org used for pushing release images (branch x.y and master)
# DOCKER_ORG_TEST, docker org used for pushing images for testing (branch testing).
env:
DOCKER_ORG: mailu
DOCKER_ORG_TEST: mailuci
jobs:
# This job calculates all global job variables that are required by all the subsequent jobs.
# All subsequent jobs will retrieve and use these variables. This way the variables only have to be derived once.
derive-variables:
name: derive variables
runs-on: ubuntu-latest
outputs:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG_DERIVED }}
BRANCH: ${{ env.BRANCH }}
DEPLOY: ${{ env.DEPLOY }}
RELEASE: ${{ env.RELEASE }}
steps:
- uses: actions/checkout@v3
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to pr-xxxx
- name: Derive MAILU_VERSION and DEPLOY/RELEASE for other branches than testing
if: env.BRANCH != 'testing'
shell: bash
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG_DERIVED=${{ env.DOCKER_ORG }}" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION and DEPLOY/RELEASE for normal release x.y
if: env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master'
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
echo "RELEASE=true" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=true" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging for master
if: env.BRANCH == 'master'
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
build-test-deploy:
needs:
- derive-variables
uses: ./.github/workflows/build_test_deploy.yml
with:
architecture: 'linux/arm64,linux/arm/v7'
mailu_version: ${{needs.derive-variables.outputs.MAILU_VERSION}}-arm
pinned_mailu_version: ${{needs.derive-variables.outputs.PINNED_MAILU_VERSION}}-arm
docker_org: ${{needs.derive-variables.outputs.DOCKER_ORG}}
branch: ${{needs.derive-variables.outputs.BRANCH}}
deploy: ${{needs.derive-variables.outputs.DEPLOY}}
release: ${{needs.derive-variables.outputs.RELEASE}}
secrets: inherit
################################################
# Code block that is used as one liner for the step:
# Derive PINNED_MAILU_VERSION and DEPLOY/RELEASE for normal release x.y
##!/bin/bash
#version=$( git tag --sort=version:refname --list "{{ env.MAILU_VERSION }}.*" | tail -1 )
#root_version=${version%.*}
#patch_version=${version##*.}
#if [ "$patch_version" == "" ]
#then
# pinned_version={{ env.MAILU_VERSION }}.0
#else
# pinned_version=$root_version.$(expr $patch_version + 1)
#fi
#echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV

355
.github/workflows/build_test_deploy.yml vendored Normal file
View File

@ -0,0 +1,355 @@
name: build-test-deploy
on:
workflow_call:
inputs:
architecture:
description: 'The architecture of the images that will be build.'
required: false
default: 'linux/amd64'
type: string
mailu_version:
description: 'The main version that is build. E.g. master or x.y.'
required: true
type: string
pinned_mailu_version:
description: 'The specific version that is build. E.g. commit hash or x.y.z.'
required: true
type: string
docker_org:
description: 'The docker organisation where the images are pushed to.'
required: true
type: string
branch:
description: 'The branch that triggered this workflow.'
required: true
type: string
deploy:
description: Deploy to docker hub. Happens for all branches but staging. Use string true or false.
default: true
required: false
type: string
release:
description: Tag and create the github release. Use string true or false.
default: false
required: false
type: string
workflow_dispatch:
inputs:
architecture:
description: 'The architecture of the images that will be build.'
required: false
default: 'linux/amd64'
type: string
mailu_version:
description: 'The main version that is build. E.g. master or x.y.'
required: true
type: string
pinned_mailu_version:
description: 'The specific version that is build. E.g. commit hash or x.y.z.'
required: true
type: string
docker_org:
description: 'The docker organisation where the images are pushed to.'
required: true
type: string
branch:
description: 'The branch that triggered this workflow.'
required: true
type: string
deploy:
description: Deploy to docker hub. Happens for all branches but staging
default: true
required: false
type: boolean
release:
description: 'Tag and create the github release. Only happens for branch x.y (release branch)'
default: false
required: false
type: boolean
env:
HCL_FILE: ./tests/build.hcl
jobs:
# This job calculates what images must be build. It reads the build.hcl file and then outputs all targets (images) in it.
# This is used by the next build job.
targets:
name: create targets
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.targets.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Create matrix
id: targets
run: |
echo ::set-output name=matrix::$(docker buildx bake -f ${{env.HCL_FILE}} --print | jq -cr '.group.default.targets')
- name: Show matrix
run: |
echo ${{ steps.targets.outputs.matrix }}
# This job builds all the images. The build cache is stored in the github actions cache.
# In further jobs, this cache is used to quickly rebuild the images.
build:
name: Build images
needs:
- targets
strategy:
fail-fast: false
matrix:
target: ${{ fromJson(needs.targets.outputs.matrix) }}
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v3
- name: Retrieve global variables
shell: bash
run: |
echo "BRANCH=${{ inputs.branch }}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.Docker_Login }}
password: ${{ secrets.Docker_Password }}
- name: Build all docker images
env:
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
uses: docker/bake-action@v2
with:
files: ${{env.HCL_FILE}}
targets: ${{ matrix.target }}
load: false
push: false
set: |
*.cache-from=type=gha,scope=build-${{ matrix.target }}
*.cache-to=type=gha,scope=build-${{ matrix.target }},mode=max
*.platform=${{ inputs.architecture }}
# This job runs all the tests.
tests:
name: tests
if: inputs.architecture == 'linux/amd64'
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
needs:
- targets
- build
strategy:
fail-fast: false
matrix:
target: ["core", "fetchmail", "filters", "snappymail", "roundcube", "webdav"]
time: ["2"]
include:
- target: "filters"
time: "3"
exclude:
- target: "filters"
time: "2"
steps:
- uses: actions/checkout@v3
- name: Retrieve global variables
shell: bash
run: |
echo "BRANCH=${{ inputs.branch }}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.Docker_Login }}
password: ${{ secrets.Docker_Password }}
- name: Build docker images for testing from cache
env:
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
uses: docker/bake-action@v2
with:
files: ${{env.HCL_FILE}}
load: true
push: false
set: |
*.cache-from=type=gha,scope=build-docs
*.cache-from=type=gha,scope=build-setup
*.cache-from=type=gha,scope=build-admin
*.cache-from=type=gha,scope=build-antispam
*.cache-from=type=gha,scope=build-front
*.cache-from=type=gha,scope=build-imap
*.cache-from=type=gha,scope=build-smtp
*.cache-from=type=gha,scope=build-snappymail
*.cache-from=type=gha,scope=build-roundcube
*.cache-from=type=gha,scope=build-antivirus
*.cache-from=type=gha,scope=build-fetchmail
*.cache-from=type=gha,scope=build-resolver
*.cache-from=type=gha,scope=build-traefik-certdumper
*.cache-from=type=gha,scope=build-webdav
*.platform=${{ inputs.architecture }}
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
- name: Copy all certs
run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*'
- name: Test ${{ matrix.target }}
run: python tests/compose/test.py ${{ matrix.target }} ${{ matrix.time }}
env:
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
# This job deploys the docker images to the docker repository. The build.hcl file contains logic that determines what tags are pushed.
# E.g. for master only the :master and :latest tags are pushed.
deploy:
name: Deploy images
# Deploying is not required for staging
if: inputs.deploy == 'true'
runs-on: ubuntu-latest
needs:
- build
- tests
steps:
- uses: actions/checkout@v3
- name: Retrieve global variables
shell: bash
run: |
echo "BRANCH=${{ inputs.branch }}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.Docker_Login }}
password: ${{ secrets.Docker_Password }}
- name: Deploy images to docker hub. Build.hcl contains the logic for the tags that are pushed.
env:
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
uses: docker/bake-action@v2
with:
files: ${{env.HCL_FILE}}
push: true
set: |
*.cache-from=type=gha,scope=build-docs
*.cache-from=type=gha,scope=build-setup
*.cache-from=type=gha,scope=build-admin
*.cache-from=type=gha,scope=build-antispam
*.cache-from=type=gha,scope=build-front
*.cache-from=type=gha,scope=build-imap
*.cache-from=type=gha,scope=build-smtp
*.cache-from=type=gha,scope=build-snappymail
*.cache-from=type=gha,scope=build-roundcube
*.cache-from=type=gha,scope=build-antivirus
*.cache-from=type=gha,scope=build-fetchmail
*.cache-from=type=gha,scope=build-resolver
*.cache-from=type=gha,scope=build-traefik-certdumper
*.cache-from=type=gha,scope=build-webdav
*.platform=${{ inputs.architecture }}
deploy-arm:
name: Deploy images for arm
# Deploying is not required for staging
if: inputs.deploy == 'true' && inputs.architecture != 'linux/amd64'
runs-on: ubuntu-latest
needs:
- build
steps:
- uses: actions/checkout@v3
- name: Retrieve global variables
shell: bash
run: |
echo "BRANCH=${{ inputs.branch }}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.Docker_Login }}
password: ${{ secrets.Docker_Password }}
- name: Deploy images to docker hub. Build.hcl contains the logic for the tags that are pushed.
env:
DOCKER_ORG: ${{ env.DOCKER_ORG }}
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
uses: docker/bake-action@v2
with:
files: ${{env.HCL_FILE}}
push: true
set: |
*.cache-from=type=gha,scope=build-docs
*.cache-from=type=gha,scope=build-setup
*.cache-from=type=gha,scope=build-admin
*.cache-from=type=gha,scope=build-antispam
*.cache-from=type=gha,scope=build-front
*.cache-from=type=gha,scope=build-imap
*.cache-from=type=gha,scope=build-smtp
*.cache-from=type=gha,scope=build-snappymail
*.cache-from=type=gha,scope=build-roundcube
*.cache-from=type=gha,scope=build-antivirus
*.cache-from=type=gha,scope=build-fetchmail
*.cache-from=type=gha,scope=build-resolver
*.cache-from=type=gha,scope=build-traefik-certdumper
*.cache-from=type=gha,scope=build-webdav
*.platform=${{ inputs.architecture }}
#This job creates a tagged release. A tag is created for the pinned version x.y.z. The GH release refers to this tag.
tag-release:
if: inputs.release == 'true'
runs-on: ubuntu-latest
needs:
- deploy
steps:
- uses: actions/checkout@v3
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Retrieve global variables
shell: bash
run: |
echo "BRANCH=${{ inputs.branch }}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Create tag for branch x.y.
shell: bash
run: |
echo git tag ${{ env.PINNED_MAILU_VERSION }} $(/usr/bin/git rev-parse HEAD)
git tag ${{ env.PINNED_MAILU_VERSION }} $(/usr/bin/git rev-parse HEAD)
git push origin ${{ env.PINNED_MAILU_VERSION }}
- name: Create release for tag x.y.z.
uses: ncipollo/release-action@v1
with:
bodyFile: "RELEASE_TEMPLATE.md"
tag: ${{ env.PINNED_MAILU_VERSION }}
token: ${{ secrets.GITHUB_TOKEN }}

134
.github/workflows/x64.yml vendored Normal file
View File

@ -0,0 +1,134 @@
name: start-linux-amd64
on:
push:
branches:
- testing
- staging
- '1.9'
- master
concurrency: ci-x64-${{ github.ref }}
###############################################
# REQUIRED secrets
# ${{ secrets.Docker_Login }}
# Username of docker login for pushing the images to repo env.DOCKER_ORG and env.DOCKER_ORG_TESTS
# ${{ secrets.Docker_Password }}
# Password of docker login for pushing the images to repo env.DOCKER_ORG and env.DOCKER_ORG_TESTS
# Add the above secrets to your github repo to determine where the images will be pushed.
################################################
# REQUIRED global variables
# DOCKER_ORG, docker org used for pushing release images (branch x.y and master)
# DOCKER_ORG_TEST, docker org used for pushing images for testing (branch testing).
env:
DOCKER_ORG: mailu
DOCKER_ORG_TEST: mailuci
jobs:
# This job calculates all global job variables that are required by all the subsequent jobs.
# All subsequent jobs will retrieve and use these variables. This way the variables only have to be derived once.
derive-variables:
name: derive variables
runs-on: ubuntu-latest
outputs:
MAILU_VERSION: ${{ env.MAILU_VERSION }}
PINNED_MAILU_VERSION: ${{ env.PINNED_MAILU_VERSION }}
DOCKER_ORG: ${{ env.DOCKER_ORG_DERIVED }}
BRANCH: ${{ env.BRANCH }}
DEPLOY: ${{ env.DEPLOY }}
RELEASE: ${{ env.RELEASE }}
steps:
- uses: actions/checkout@v3
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Extract branch name
shell: bash
run: |
echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
#For branch TESTING, we set the image tag to pr-xxxx
- name: Derive MAILU_VERSION and PINNED_MAILU_VERSION and DEPLOY/RELEASE for branch testing
if: env.BRANCH == 'testing'
shell: bash
env:
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
run: |
echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV
echo "DOCKER_ORG_DERIVED=${{ env.DOCKER_ORG_TEST }}" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
- name: Derive MAILU_VERSION and DEPLOY/RELEASE for other branches than testing
if: env.BRANCH != 'testing'
shell: bash
run: |
echo "MAILU_VERSION=${{ env.BRANCH }}" >> $GITHUB_ENV
echo "DOCKER_ORG_DERIVED=${{ env.DOCKER_ORG }}" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION and DEPLOY/RELEASE for normal release x.y
if: env.BRANCH != 'testing' && env.BRANCH != 'staging' && env.BRANCH != 'master'
shell: bash
run: |
version=$( git tag --sort=version:refname --list "${{ env.MAILU_VERSION }}.*" | tail -1 );root_version=${version%.*};patch_version=${version##*.};if [ "$patch_version" == "" ]; then pinned_version=${{ env.MAILU_VERSION }}.0; else pinned_version=$root_version.$(expr $patch_version + 1); fi;echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV
echo "RELEASE=true" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=true" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION and DEPLOY/RELEASE for staging
id: deploy
if: env.BRANCH == 'staging'
shell: bash
run: |
echo "PINNED_MAILU_VERSION=staging" >> $GITHUB_ENV
echo "DEPLOY=false" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
- name: Derive PINNED_MAILU_VERSION for staging for master
if: env.BRANCH == 'master'
shell: bash
env:
GITHUB_SHA: ${{ env.GITHUB_SHA }}
run: |
echo "PINNED_MAILU_VERSION=$GITHUB_SHA" >> $GITHUB_ENV
echo "DEPLOY=true" >> $GITHUB_ENV
echo "RELEASE=false" >> $GITHUB_ENV
build-test-deploy:
needs:
- derive-variables
uses: ./.github/workflows/build_test_deploy.yml
with:
architecture: 'linux/amd64'
mailu_version: ${{needs.derive-variables.outputs.MAILU_VERSION}}
pinned_mailu_version: ${{needs.derive-variables.outputs.PINNED_MAILU_VERSION}}
docker_org: ${{needs.derive-variables.outputs.DOCKER_ORG}}
branch: ${{needs.derive-variables.outputs.BRANCH}}
deploy: ${{needs.derive-variables.outputs.DEPLOY}}
release: ${{needs.derive-variables.outputs.RELEASE}}
secrets: inherit
# This job is watched by bors. It only complets if building,testing and deploy worked.
ci-success:
name: CI-Done
#Returns true when none of the **previous** steps have failed or have been canceled.
if: success()
needs:
- build-test-deploy
runs-on: ubuntu-latest
steps:
- name: CI/CD succeeded.
run: exit 0
################################################
# Code block that is used as one liner for the step:
# Derive PINNED_MAILU_VERSION and DEPLOY/RELEASE for normal release x.y
##!/bin/bash
#version=$( git tag --sort=version:refname --list "{{ env.MAILU_VERSION }}.*" | tail -1 )
#root_version=${version%.*}
#patch_version=${version##*.}
#if [ "$patch_version" == "" ]
#then
# pinned_version={{ env.MAILU_VERSION }}.0
#else
# pinned_version=$root_version.$(expr $patch_version + 1)
#fi
#echo "PINNED_MAILU_VERSION=$pinned_version" >> $GITHUB_ENV

4
core/admin/Dockerfile
View File

@ -1,8 +1,7 @@
# First stage to build assets # First stage to build assets
ARG DISTRO=alpine:3.14.5 ARG DISTRO=alpine:3.14.5
ARG ARCH=""
FROM ${ARCH}node:16 as assets FROM node:16-alpine3.16 as assets
COPY package.json ./ COPY package.json ./
RUN set -eu \ RUN set -eu \
@ -22,7 +21,6 @@ RUN set -eu \
# Actual application # Actual application
FROM $DISTRO FROM $DISTRO
ARG VERSION ARG VERSION
COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static
ENV TZ Etc/UTC ENV TZ Etc/UTC

4
core/dovecot/Dockerfile
View File

@ -15,7 +15,9 @@ RUN apk add --no-cache \
RUN pip3 install socrate==0.2.0 RUN pip3 install socrate==0.2.0
# Shared layer between dovecot and postfix # Shared layer between dovecot and postfix
RUN pip3 install "podop>0.2.5" RUN apk add --no-cache --virtual .build-deps gcc musl-dev python3-dev \
&& pip3 install "podop>0.2.5" \
&& apk del .build-deps
# Image specific layers under this line # Image specific layers under this line
RUN apk add --no-cache \ RUN apk add --no-cache \

4
core/postfix/Dockerfile
View File

@ -16,7 +16,9 @@ RUN apk add --no-cache \
RUN pip3 install socrate==0.2.0 RUN pip3 install socrate==0.2.0
# Shared layer between dovecot and postfix # Shared layer between dovecot and postfix
RUN pip3 install "podop>0.2.5" RUN apk add --no-cache --virtual .build-deps gcc musl-dev python3-dev \
&& pip3 install "podop>0.2.5" \
&& apk del .build-deps
# Image specific layers under this line # Image specific layers under this line
# Building pycares from source requires py3-wheel and libffi-dev packages # Building pycares from source requires py3-wheel and libffi-dev packages

76
docs/arm_images.rst Normal file
View File

@ -0,0 +1,76 @@
.. _arm_images:
Arm images for Mailu
====================
Using Mailu arm images
----------------------
The Mailu project makes use of github actions for automatic CI/CD.
Github actions only has x64 (amd64) runners. This means we can build the arm images
using QEMU, but we cannot test the images. For this reason the arm images have
a BETA status. We only guarantee that the images could be built.
We strongly recommend to make use of the pinned version (tag 1.9.10 vs tag 1.9).
Pinned versions (tag x.y.z) are not updated. This allows upgrading manually by changing the
tag to the next pinned version.
Whenever images are deployed for master and for releases (branch x.y),
images are also built for arm.
The images are pushed with -arm appended to the tag. For example:
- admin:master-arm
- admin:1.10-arm
To use these images, simply use setup.mailu.io for generating the docker-composse.yml
file and mailu.env file. Then in the docker-compose.yml file append -arm to the tags of
all images from the mailu docker repository.
Build manually
--------------
It is possible to build the images manually. There are two possiblities for this.
Github actions
``````````````
The main workflow build-test-deploy can be triggered manually.
Via the parameter ``architecture`` the target platform can be specified.
Use the value ``'linux/arm64,linux/arm/v7'``.
To use it:
1. Fork the Mailu github project.
2. In the settings of your forked project, configure the secrets Docker_Login and Docker_Password. For more information on these secrets, see the comments in the build-test-deploy.yml file.
3. In the forked project, trigger the workflow build-test-deploy manually.
4. For the parameter architecture use the value ``'linux/arm64,linux/arm/v7'``.
Manually
````````
It is also possible to build the images manually on bare-metal.
The buildx file ``tests/build.hcl`` can be used for this.
To build manually:
1. Install QEMU static binaries. This is only required, if you don't built on an arm machine. For Ubuntu install qemu-user-static.
2. Clone the Mailu github project.
3. Export the parameters.
4. Create a buildx builder instance
5. Run buildx overriding the architecture.
For example:
.. code-block:: bash
docker login
Username: Foo
Password: Bar
export DOCKER_ORG="Foo"
export MAILU_VERSION="master-arm"
export MAILU_PINNED_VERSION="hash"
docker buildx create --use
docker buildx bake -f tests/build.hcl --push --set *.platform=linux/arm64,linux/arm/v7

23
docs/contributors/environment.rst
View File

@ -106,26 +106,29 @@ The development environment is quite similar to the production one.
Building images Building images
``````````````` ```````````````
We supply a separate ``test/build.yml`` file for convenience. We supply a separate ``test/build.hcl`` file for convenience.
After cloning the git repository to your workstation, you can build the images: After cloning the git repository to your workstation, you can build the images:
.. code-block:: bash .. code-block:: bash
cd Mailu cd Mailu
docker-compose -f tests/build.yml build docker buildx bake -f tests/build.hcl --load
The ``build.yml`` file has two variables: The ``build.hcl`` file has three variables:
#. ``$DOCKER_ORG``: First part of the image tag. Defaults to *mailu* and needs to be changed #. ``$DOCKER_ORG``: First part of the image tag. Defaults to *mailu* and needs to be changed
only when pushing to your own Docker hub account. only when pushing to your own Docker hub account.
#. ``$VERSION``: Last part of the image tag. Defaults to *local* to differentiate from pulled #. ``$MAILU_VERSION``: Last part of the image tag. Defaults to *local* to differentiate from pulled
images. images.
#. ``$MAILU_PINNED_VERSION``: Last part of the image tag for x.y.z images. Defaults to *local* to differentiate from pulled
images.
To re-build only specific containers at a later time. To re-build only specific containers at a later time.
.. code-block:: bash .. code-block:: bash
docker-compose -f tests/build.yml build admin webdav docker buildx bake -f tests/build.hcl admin webdav
If you have to push the images to Docker Hub for testing in Docker Swarm or a remote If you have to push the images to Docker Hub for testing in Docker Swarm or a remote
host, you have to define ``DOCKER_ORG`` (usually your Docker user-name) and login to host, you have to define ``DOCKER_ORG`` (usually your Docker user-name) and login to
@ -137,15 +140,15 @@ the hub.
Username: Foo Username: Foo
Password: Bar Password: Bar
export DOCKER_ORG="Foo" export DOCKER_ORG="Foo"
export VERSION="feat-extra-app" export MAILU_VERSION="feat-extra-app"
docker-compose -f tests/build.yml build export MAILU_PINNED_VERSION="feat-extra-app"
docker-compose -f tests/build.yml push docker buildx bake -f tests/build.hcl --push
Running containers Running containers
`````````````````` ``````````````````
To run the newly created images: ``cd`` to your project directory. Edit ``.env`` to set To run the newly created images: ``cd`` to your project directory. Edit ``.env`` to set
``VERSION`` to the same value as used during the build, which defaults to ``local``. ``VERSION`` to the same value as used during the build (for MAILU_VERSION), which defaults to ``local``.
After that you can run: After that you can run:
.. code-block:: bash .. code-block:: bash
@ -277,7 +280,7 @@ Merge conflicts
Before proceeding, check the PR page in the bottom. It should not indicate a merge conflict. Before proceeding, check the PR page in the bottom. It should not indicate a merge conflict.
If there are merge conflicts, you have 2 options: If there are merge conflicts, you have 2 options:
#. Do a review "request changes" and ask the author to resolve the merge conflict. #. Do a review "request changes" and ask the author to resolve the merge conflict.
#. Solve the merge conflict yourself on Github, using the web editor. #. Solve the merge conflict yourself on Github, using the web editor.
If it can't be done in the web editor, go for option 1. Unless you want to go through the trouble of If it can't be done in the web editor, go for option 1. Unless you want to go through the trouble of

2
docs/index.rst
View File

@ -60,7 +60,7 @@ the version of Mailu that you are running.
dns dns
reverse reverse
database database
rpi_build arm_images
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2

25
docs/rpi_build.rst
View File

@ -1,25 +0,0 @@
.. _rpi_build:
Building for a Raspberry Pi
===========================
The build does not need to be done on the Pi.
To do so, go to ``tests/`` and call ``./build_arm.sh``, it will build all
necessary images for arm.
To push the locally built images to a remote server, run ``./deploy_to_pi.sh``.
Docker 18.09+ is needed to use ``-H ssh://<user>@<host>``.
Adjustments
-----------
``build_arm.sh`` uses some variables passed as ``build-arg`` to docker-compose:
- ``ALPINE_VER``: version of ALPINE to use
- ``DISTRO``: is the main distro used. Dockerfiles are set on Alpine 3.14, and
build script overrides for ``balenalib/rpi-alpine:3.14``
- ``QEMU``: Used by webmails dockerfiles. It will add ``qemu-arm-static`` only
if ``QEMU`` is set to ``arm``
- ``ARCH``: Architecture to use for ``admin``, and ``webmails`` as their images
are available for those architectures.

2
docs/setup.rst
View File

@ -19,7 +19,7 @@ Prepare the environment
Mailu images are designed to work on x86 or equivalent hardware, so it Mailu images are designed to work on x86 or equivalent hardware, so it
should run on pretty much any cloud server as long as enough power is should run on pretty much any cloud server as long as enough power is
provided. For non x86 machines, see :ref:`rpi_build` provided. For non x86 machines, see :ref:`arm_images`
You are free to choose any operating system that runs Docker (>= 1.11), You are free to choose any operating system that runs Docker (>= 1.11),
then chose between various flavors including Docker Compose, Kubernetes then chose between various flavors including Docker Compose, Kubernetes

185
tests/build.hcl Normal file
View File

@ -0,0 +1,185 @@
# build.hcl
# For more information on buildx bake file definition see:
# https://github.com/docker/buildx/blob/master/docs/guides/bake/file-definition.md
#
# NOTE: You can only run this from the Mailu root folder.
# Make sure the context is Mailu (project folder) and not Mailu/tests
#-----------------------------------------------------------------------------------------
# (Environment) input variables
# If the env var is not set, then the default value is used
#-----------------------------------------------------------------------------------------
variable "DOCKER_ORG" {
default = "mailu"
}
variable "DOCKER_PREFIX" {
default = ""
}
variable "PINNED_MAILU_VERSION" {
default = "local"
}
variable "MAILU_VERSION" {
default = "local"
}
#-----------------------------------------------------------------------------------------
# Grouping of targets to build. All these images are built when using:
# docker buildx bake -f tests\build.hcl
#-----------------------------------------------------------------------------------------
group "default" {
targets = [
"docs",
"setup",
"admin",
"antispam",
"front",
"imap",
"smtp",
"snappymail",
"roundcube",
"antivirus",
"fetchmail",
"resolver",
"traefik-certdumper",
"webdav"
]
}
#-----------------------------------------------------------------------------------------
# Default settings that will be inherited by all targets (images to build).
#-----------------------------------------------------------------------------------------
target "defaults" {
platforms = [ "linux/amd64"]
dockerfile = "Dockerfile"
args = {
VERSION = "${PINNED_MAILU_VERSION}"
}
}
#-----------------------------------------------------------------------------------------
# User defined functions
#------------------------------------------------------------------------------------------
# Derive all tags
function "tag" {
params = [image_name]
result = [ notequal("master",MAILU_VERSION) && notequal("master-arm",MAILU_VERSION) ? "${DOCKER_ORG}/${DOCKER_PREFIX}${image_name}:${PINNED_MAILU_VERSION}": "",
"${DOCKER_ORG}/${DOCKER_PREFIX}${image_name}:${MAILU_VERSION}",
"${DOCKER_ORG}/${DOCKER_PREFIX}${image_name}:latest"
]
}
#-----------------------------------------------------------------------------------------
# All individual targets (images to build)
# Build an individual target using.
# docker buildx bake -f tests\build.hcl <target>
# E.g. to build target docs
# docker buildx bake -f tests\build.hcl docs
#-----------------------------------------------------------------------------------------
# -----------------------------------------------------------------------------------------
# Documentation and setup images
# -----------------------------------------------------------------------------------------
target "docs" {
inherits = ["defaults"]
context = "docs"
tags = tag("docs")
args = {
version = "${MAILU_VERSION}"
pinned_version = "${PINNED_MAILU_VERSION}"
}
}
target "setup" {
inherits = ["defaults"]
context="setup"
tags = tag("setup")
}
# -----------------------------------------------------------------------------------------
# Core images
# -----------------------------------------------------------------------------------------
target "none" {
inherits = ["defaults"]
context="core/none"
tags = tag("none")
}
target "admin" {
inherits = ["defaults"]
context="core/admin"
tags = tag("admin")
}
target "antispam" {
inherits = ["defaults"]
context="core/rspamd"
tags = tag("rspamd")
}
target "front" {
inherits = ["defaults"]
context="core/nginx"
tags = tag("nginx")
}
target "imap" {
inherits = ["defaults"]
context="core/dovecot"
tags = tag("dovecot")
}
target "smtp" {
inherits = ["defaults"]
context="core/postfix"
tags = tag("postfix")
}
# -----------------------------------------------------------------------------------------
# Webmail images
# -----------------------------------------------------------------------------------------
target "snappymail" {
inherits = ["defaults"]
context="webmails/snappymail"
tags = tag("snappymail")
}
target "roundcube" {
inherits = ["defaults"]
context="webmails/roundcube"
tags = tag("roundcube")
}
# -----------------------------------------------------------------------------------------
# Optional images
# -----------------------------------------------------------------------------------------
target "antivirus" {
inherits = ["defaults"]
context="optional/clamav"
tags = tag("clamav")
}
target "fetchmail" {
inherits = ["defaults"]
context="optional/fetchmail"
tags = tag("fetchmail")
}
target "resolver" {
inherits = ["defaults"]
context="optional/unbound"
tags = tag("unbound")
}
target "traefik-certdumper" {
inherits = ["defaults"]
context="optional/traefik-certdumper"
tags = tag("traefik-certdumper")
}
target "webdav" {
inherits = ["defaults"]
context="optional/radicale"
tags = tag("radicale")
}

109
tests/build.yml
View File

@ -1,109 +0,0 @@
version: '3'
services:
front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/nginx
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
resolver:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}unbound:${PINNED_MAILU_VERSION:-local}
build:
context: ../optional/unbound
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/dovecot
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/postfix
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/rspamd
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
antivirus:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}clamav:${PINNED_MAILU_VERSION:-local}
build:
context: ../optional/clamav
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
webdav:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}radicale:${PINNED_MAILU_VERSION:-local}
build:
context: ../optional/radicale
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
traefik-certdumper:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}traefik-certdumper:${PINNED_MAILU_VERSION:-local}
build:
context: ../optional/traefik-certdumper
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/admin
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
roundcube:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}roundcube:${PINNED_MAILU_VERSION:-local}
build:
context: ../webmails/roundcube
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
snappymail:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}snappymail:${PINNED_MAILU_VERSION:-local}
build:
context: ../webmails/snappymail
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
fetchmail:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${PINNED_MAILU_VERSION:-local}
build:
context: ../optional/fetchmail
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
none:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}none:${PINNED_MAILU_VERSION:-local}
build:
context: ../core/none
args:
VERSION: ${PINNED_MAILU_VERSION:-local}
docs:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}docs:${PINNED_MAILU_VERSION:-local}
build:
context: ../docs
args:
version: ${MAILU_VERSION:-local}
pinned_version: ${PINNED_MAILU_VERSION:-local}
setup:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}setup:${PINNED_MAILU_VERSION:-local}
build:
context: ../setup
args:
VERSION: ${PINNED_MAILU_VERSION:-local}

15
tests/build_arm.sh
View File

@ -1,15 +0,0 @@
#!/bin/bash -x
ALPINE_VER="3.14"
DISTRO="balenalib/rpi-alpine:$ALPINE_VER"
# Used for webmails
QEMU="arm"
ARCH="arm32v7/"
# use qemu-*-static from docker container
docker run --rm --privileged multiarch/qemu-user-static:register
docker-compose -f build.yml build \
--build-arg DISTRO=$DISTRO \
--build-arg ARCH=$ARCH \
--build-arg QEMU=$QEMU \
--parallel $@

10
tests/compose/core/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- resolver - resolver
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:

12
tests/compose/fetchmail/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- 192.168.203.254 - 192.168.203.254
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -80,7 +80,7 @@ services:
fetchmail: fetchmail:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env

12
tests/compose/filters/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- resolver - resolver
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -78,7 +78,7 @@ services:
# Optional services # Optional services
antivirus: antivirus:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}clamav:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}clamav:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:

12
tests/compose/roundcube/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- 192.168.203.254 - 192.168.203.254
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -88,7 +88,7 @@ services:
# Webmail # Webmail
webmail: webmail:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}roundcube:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}roundcube:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:

12
tests/compose/snappymail/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- 192.168.203.254 - 192.168.203.254
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -88,7 +88,7 @@ services:
# Webmail # Webmail
webmail: webmail:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}snappymail:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}snappymail:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:

12
tests/compose/webdav/docker-compose.yml
View File

@ -15,7 +15,7 @@ services:
# Core services # Core services
front: front:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
logging: logging:
@ -34,7 +34,7 @@ services:
- "/mailu/certs:/certs" - "/mailu/certs:/certs"
admin: admin:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -47,7 +47,7 @@ services:
- resolver - resolver
imap: imap:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -57,7 +57,7 @@ services:
- front - front
smtp: smtp:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -66,7 +66,7 @@ services:
- front - front
antispam: antispam:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:
@ -79,7 +79,7 @@ services:
# Optional services # Optional services
webdav: webdav:
image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}radicale:${PINNED_MAILU_VERSION:-local} image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}radicale:${MAILU_VERSION:-local}
restart: always restart: always
env_file: mailu.env env_file: mailu.env
volumes: volumes:

51
tests/deploy.sh
View File

@ -1,51 +0,0 @@
#!/bin/bash
# Skip deploy for staging branch
[ "$BRANCH" = "staging" ] && exit 0
docker login -u $DOCKER_UN -p $DOCKER_PW
if [ "$BRANCH" = "testing" ]
then
docker-compose -f tests/build.yml push
exit 0
fi
#Deploy for main releases
#Images are built with tag PINNED_MAILU_VERSION (x.y.z).
#We are tagging them as well with MAILU_VERSION (x.y)
#After that, both tags are pushed to the docker repository.
if [ "$PINNED_MAILU_VERSION" != "" ] && [ "$BRANCH" != "master" ]
then
images=$(docker-compose -f tests/build.yml config | awk -F ':' '/image:/{ print $2 }')
for image in $images
do
docker tag "${image}":"${PINNED_MAILU_VERSION}" "${image}":${MAILU_VERSION}
done
#Push PINNED_MAILU_VERSION images
docker-compose -f tests/build.yml push
#Push MAILU_VERSION images
PINNED_MAILU_VERSION=$MAILU_VERSION
docker-compose -f tests/build.yml push
exit 0
fi
#Deploy for master. For master we only publish images with tag master
#Images are built with tag PINNED_MAILU_VERSION (commit hash).
#We are tagging them as well with MAILU_VERSION (master)
#Then we publish the images with tag master
if [ "$PINNED_MAILU_VERSION" != "" ] && [ "$BRANCH" == "master" ]
then
images=$(docker-compose -f tests/build.yml config | awk -F ':' '/image:/{ print $2 }')
for image in $images
do
docker tag "${image}":"${PINNED_MAILU_VERSION}" "${image}":${MAILU_VERSION}
done
#Push MAILU_VERSION images
PINNED_MAILU_VERSION=$MAILU_VERSION
docker-compose -f tests/build.yml push
exit 0
fi
#Fallback in case $PINNED_MAILU_VERSION is empty. This should never execute.
docker-compose -f tests/build.yml push

6
tests/deploy_to_pi.sh
View File

@ -1,6 +0,0 @@
#!/bin/bash -x
# Small script to deploy locally built images to a remote docker
compose_options=$1
images=$(docker-compose $1 images | awk 'NR > 2 { printf $2":"$3" " }')
docker save $images | pigz - > mail.local.tgz
echo "now run 'docker -H \"ssh://user@host\" load -i mail.local.tgz"

9
towncrier/newsfragments/2383.misc Normal file
View File

@ -0,0 +1,9 @@
Switch from docker build to buildx for CI/CD.
- The main workflow file has been optimised and simplified.
- Images are built in parallel when building locally resulting in faster build times.
- The github action workflow is about 50% faster.
- Arm images are built as well. These images are not tested due to restrictions of github actions (no arm runners). The tags of the images have -arm appended to it.
- Arm images can also be built locally.
- Reusable workflow is introduced for building, testing and deploying the images.
This allows the workflow to be reused for other purposes in the future.
- Workflow can be manually triggered. This allows forked Mailu projects to also use the workflow for building images.

8
webmails/roundcube/Dockerfile
View File

@ -1,12 +1,6 @@
# NOTE: only add qemu-arm-static if building for arm # NOTE: only add qemu-arm-static if building for arm
ARG ARCH="" FROM php:8.0-apache
ARG QEMU=other
FROM ${ARCH}php:8.0-apache as build_arm
ONBUILD COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static
FROM ${ARCH}php:8.0-apache as build_other
FROM build_${QEMU}
ARG VERSION ARG VERSION
ENV TZ Etc/UTC ENV TZ Etc/UTC