Single binary executables (#1330)

* Single binary executables / DEB packages.

* Attach single binary executables to the releases

extras/packaging/linux/Dockerfile

@@ -0,0 +1,32 @@
+# Use the oldest (but still supported) Ubuntu as the base for PyInstaller
+# packages. This will prevent stuff like glibc from conflicting.
+FROM ubuntu:18.04
+
+RUN apt-get update
+RUN apt-get install -y software-properties-common binutils
+RUN apt-get install -y ruby-dev
+RUN gem install fpm
+
+# Use deadsnakes for the latest Pythons (e.g 3.9)
+RUN add-apt-repository ppa:deadsnakes/ppa
+RUN apt-get update && apt-get install -y python3.9 python3.9-dev python3.9-venv
+
+# Install rpm as well, since we are going to build fedora dists too
+RUN apt-get install -y rpm
+
+ADD . /app
+WORKDIR /app/extras/packaging/linux
+
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3.9 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+# Ensure that pip is renewed, otherwise we would be using distro-provided pip
+# which strips vendored packages and doesn't work with PyInstaller.
+RUN python -m pip install /app
+RUN python -m pip install pyinstaller wheel
+RUN python -m pip install --force-reinstall --upgrade pip
+
+RUN python build.py
+
+ENTRYPOINT ["mv", "/app/extras/packaging/linux/dist/", "/artifacts"]

extras/packaging/linux/README.md

@@ -0,0 +1,52 @@
+# Standalone Linux Packages
+
+![packaging.png](https://user-images.githubusercontent.com/47358913/159950478-2d090d1b-69b9-4914-a1b4-d3e3d8e25fe0.png)
+
+This directory contains the build scripts for creating:
+
+- A self-contained binary executable for the HTTPie itself
+- `httpie.deb` and `httpie.rpm` packages for Debian and Fedora.
+
+The process of constructing them are fully automated, and can be easily done through the [`Release as Standalone Linux Package`](https://github.com/httpie/httpie/actions/workflows/release-linux-standalone.yml)
+action. Once it finishes, the release artifacts will be attached in the summary page of the triggered run.
+
+
+## Hacking
+
+The main entry point for the package builder is the [`build.py`](https://github.com/httpie/httpie/blob/master/extras/packaging/linux/build.py). It
+contains 2 major methods:
+
+- `build_binaries`, for the self-contained executables
+- `build_packages`, for the OS-specific packages (which wrap the binaries)
+
+### `build_binaries`
+
+We use [PyInstaller](https://pyinstaller.readthedocs.io/en/stable/) for the binaries. Normally pyinstaller offers two different modes:
+
+- Single directory (harder to distribute, low redundancy. Library files are shared accross different executables)
+- Single binary (easier to distribute, higher redundancy. Same libraries are statically linked to different executables, so higher total size)
+
+Since our binary size (in total 20 MiBs) is not that big, we have decided to choose the single binary mode for the sake of easier distribution.
+
+We also disable `UPX`, which is a runtime decompression method since it adds some startup cost.
+
+### `build_packages`
+
+We build our OS-specific packages with [FPM](https://github.com/jordansissel/fpm) which offers a really nice abstraction. We use the `dir` mode,
+and package `http`, `https` and `httpie` commands. More can be added to the `files` option.
+
+Since the `httpie` depends on having a pip executable, we explicitly depend on the system Python even though the core does not use it.
+
+### Docker Image
+
+This directory also contains a [docker image](https://github.com/httpie/httpie/blob/master/extras/packaging/linux/Dockerfile) which helps
+building our standalone binaries in an isolated environment with the lowest possible library versions. This is important, since even though
+the executables are standalone they still depend on some main system C libraries (like `glibc`) so we need to create our executables inside
+an environment with a very old (but not deprecated) glibc version. It makes us soundproof for all active Ubuntu/Debian versions.
+
+It also contains the Python version we package our HTTPie with, so it is the place if you need to change it.
+
+### `./get_release_artifacts.sh`
+
+If you make a change in the `build.py`, run the following script to test it out. It will return multiple files under `artifacts/dist` which
+then you can test out and ensure their quality (it is also the script that we use in our automation).

extras/packaging/linux/build.py

@@ -0,0 +1,100 @@
+import stat
+import subprocess
+from pathlib import Path
+from typing import Iterator, Tuple
+
+BUILD_DIR = Path(__file__).parent
+HTTPIE_DIR = BUILD_DIR.parent.parent.parent
+
+SCRIPT_DIR = BUILD_DIR / Path('scripts')
+HOOKS_DIR = SCRIPT_DIR / 'hooks'
+
+DIST_DIR = BUILD_DIR / 'dist'
+
+TARGET_SCRIPTS = {
+    SCRIPT_DIR / 'http_cli.py': [],
+    SCRIPT_DIR / 'httpie_cli.py': ['--hidden-import=pip'],
+}
+
+
+def build_binaries() -> Iterator[Tuple[str, Path]]:
+    for target_script, extra_args in TARGET_SCRIPTS.items():
+        subprocess.check_call(
+            [
+                'pyinstaller',
+                '--onefile',
+                '--noupx',
+                '-p',
+                HTTPIE_DIR,
+                '--additional-hooks-dir',
+                HOOKS_DIR,
+                *extra_args,
+                target_script,
+            ]
+        )
+
+    for executable_path in DIST_DIR.iterdir():
+        if executable_path.suffix:
+            continue
+        stat_r = executable_path.stat()
+        executable_path.chmod(stat_r.st_mode | stat.S_IEXEC)
+        yield executable_path.stem, executable_path
+
+
+def build_packages(http_binary: Path, httpie_binary: Path) -> None:
+    import httpie
+
+    # Mapping of src_file -> dst_file
+    files = [
+        (http_binary, '/usr/bin/http'),
+        (http_binary, '/usr/bin/https'),
+        (httpie_binary, '/usr/bin/httpie'),
+    ]
+    # A list of additional dependencies
+    deps = [
+        'python3 >= 3.7',
+        'python3-pip'
+    ]
+
+    processed_deps = [
+        f'--depends={dep}'
+        for dep in deps
+    ]
+    processed_files = [
+        '='.join([str(src.resolve()), dst]) for src, dst in files
+    ]
+    for target in ['deb', 'rpm']:
+        subprocess.check_call(
+            [
+                'fpm',
+                '--force',
+                '-s',
+                'dir',
+                '-t',
+                target,
+                '--name',
+                'httpie',
+                '--version',
+                httpie.__version__,
+                '--description',
+                httpie.__doc__.strip(),
+                '--license',
+                httpie.__licence__,
+                *processed_deps,
+                *processed_files,
+            ],
+            cwd=DIST_DIR,
+        )
+
+
+def main():
+    binaries = dict(build_binaries())
+    build_packages(binaries['http_cli'], binaries['httpie_cli'])
+
+    # Rename http_cli/httpie_cli to http/httpie
+    binaries['http_cli'].rename('http')
+    binaries['httpie_cli'].rename('httpie')
+
+
+if __name__ == '__main__':
+    main()

extras/packaging/linux/get_release_artifacts.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -xe
+
+REPO_ROOT=../../../
+ARTIFACTS_DIR=$(pwd)/artifacts
+
+# Reset the ARTIFACTS_DIR.
+rm -rf $ARTIFACTS_DIR
+mkdir -p $ARTIFACTS_DIR
+
+# Operate on the repository root to have the proper
+# docker context.
+pushd $REPO_ROOT
+
+# Build the PyInstaller image
+docker build -t pyinstaller-httpie -f extras/packaging/linux/Dockerfile .
+
+# Copy the artifacts to the designated directory.
+docker run --rm -i -v $ARTIFACTS_DIR:/artifacts pyinstaller-httpie:latest
+
+popd

extras/packaging/linux/scripts/hooks/hook-pip.py

@@ -0,0 +1,14 @@
+from pathlib import Path
+from PyInstaller.utils.hooks import collect_all
+
+def hook(hook_api):
+    for pkg in [
+        'pip',
+        'setuptools',
+        'distutils',
+        'pkg_resources'
+    ]:
+        datas, binaries, hiddenimports = collect_all(pkg)
+        hook_api.add_datas(datas)
+        hook_api.add_binaries(binaries)
+        hook_api.add_imports(*hiddenimports)

extras/packaging/linux/scripts/http_cli.py

@@ -0,0 +1,5 @@
+from httpie.__main__ import main
+
+if __name__ == '__main__':
+    import sys
+    sys.exit(main())

extras/packaging/linux/scripts/httpie_cli.py

@@ -0,0 +1,5 @@
+from httpie.manager.__main__ import main
+
+if __name__ == '__main__':
+    import sys
+    sys.exit(main())