mirror of
https://github.com/pgbackrest/pgbackrest.git
synced 2024-12-12 10:04:14 +02:00
Add building a development environment to contributing documentation.
This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros. Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI. We'll address this in the future when we add contributing documentation to the website.
This commit is contained in:
David Steele
parent
8b682b75d2
commit
eca00e0be0
159
CONTRIBUTING.md
159
CONTRIBUTING.md
@ -4,6 +4,165 @@
|
||||
|
||||
This documentation is intended to assist contributors to pgBackRest by outlining some basic steps and guidelines for contributing to the project. Coding standards to follow are defined in [CODING.md](https://github.com/pgbackrest/pgbackrest/blob/master/CODING.md). At a minimum, unit tests must be written and run and the documentation generated before submitting a Pull Request; see the [Testing](#testing) section below for details.
|
||||
|
||||
## Building a Development Environment
|
||||
|
||||
This example is based on Ubuntu 19.04, but it should work on many versions of Debian and Ubuntu.
|
||||
|
||||
pgbackrest-dev => Install development tools
|
||||
```
|
||||
sudo apt-get install rsync git devscripts build-essential valgrind autoconf \
|
||||
libssl-dev zlib1g-dev libperl-dev libxml2-dev libpq-dev libxml-checker-perl \
|
||||
libyaml-libyaml-perl libdbd-pg-perl
|
||||
```
|
||||
|
||||
Coverage testing is an important component of pgBackRest testing and is integrated directly into the test harness. Unfortunately, the default version of lcov is often not compatible with gcc. lcov 1.14 works up to gcc 8.
|
||||
|
||||
pgbackrest-dev => Build lcov 2.14
|
||||
```
|
||||
curl -fsSL \
|
||||
https://github.com/linux-test-project/lcov/releases/download/v1.14/lcov-1.14.tar.gz | \
|
||||
tar zx
|
||||
sudo make -C lcov-1.14 install
|
||||
```
|
||||
|
||||
Some unit tests and all the integration test require Docker. Running in containers allows us to simulate multiple hosts, test on different distributions and versions of PostgreSQL, and use sudo without affecting the host system.
|
||||
|
||||
pgbackrest-dev => Install Docker
|
||||
```
|
||||
curl -fsSL https://get.docker.com | sudo sh
|
||||
sudo usermod -aG docker `whoami`
|
||||
```
|
||||
|
||||
This clone of the pgBackRest repository is sufficient for experimentation. For development, create a fork and clone that instead.
|
||||
|
||||
pgbackrest-dev => Clone pgBackRest repository
|
||||
```
|
||||
git clone https://github.com/pgbackrest/pgbackrest.git
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Without Docker
|
||||
|
||||
If Docker is not installed, then the available tests can be listed using `--vm-none`.
|
||||
|
||||
pgbackrest-dev => List tests that don't require a container
|
||||
```
|
||||
pgbackrest/test/test.pl --vm=none --dry-run
|
||||
|
||||
--- output ---
|
||||
|
||||
[filtered 2 lines of output]
|
||||
2019-10-30 10:16:44.828 P00 INFO: check version info
|
||||
2019-10-30 10:16:45.318 P00 INFO: builds required: bin
|
||||
--> 2019-10-30 10:16:45.319 P00 INFO: 64 tests selected
|
||||
|
||||
2019-10-30 10:16:45.319 P00 INFO: P1-T01/64 - vm=none, module=common, test=error
|
||||
[filtered 61 lines of output]
|
||||
2019-10-30 10:16:45.334 P00 INFO: P1-T63/64 - vm=none, module=command, test=storage
|
||||
2019-10-30 10:16:45.334 P00 INFO: P1-T64/64 - vm=none, module=performance, test=type
|
||||
--> 2019-10-30 10:16:45.389 P00 INFO: DRY RUN COMPLETED SUCCESSFULLY (1s)
|
||||
```
|
||||
|
||||
Once a test has been selected it can be run by specifying the module and test. The `--dev` option sets several flags that are appropriate for development rather than test. Most importantly, it reuses object files from the previous test run to speed testing. The `--vm-out` option displays the test output.
|
||||
|
||||
pgbackrest-dev => Run a test
|
||||
```
|
||||
pgbackrest/test/test.pl --vm=none --dev --vm-out --module=common --test=wait
|
||||
|
||||
--- output ---
|
||||
|
||||
2019-10-30 10:16:46.166 P00 INFO: test begin - log level info
|
||||
2019-10-30 10:16:46.743 P00 INFO: check code autogenerate
|
||||
2019-10-30 10:16:46.743 P00 INFO: cleanup old data
|
||||
2019-10-30 10:16:47.278 P00 INFO: builds required: none
|
||||
2019-10-30 10:16:47.282 P00 INFO: 1 test selected
|
||||
|
||||
2019-10-30 10:16:47.283 P00 INFO: P1-T1/1 - vm=none, module=common, test=wait
|
||||
|
||||
run 001 - waitNew(), waitMore, and waitFree()
|
||||
000.004s l0018 - expect AssertError: assertion 'waitTime >= 100 && waitTime <= 999999000' failed
|
||||
000.020s l0019 - expect AssertError: assertion 'waitTime >= 100 && waitTime <= 999999000' failed
|
||||
000.024s l0024 - new wait = 0.2 sec
|
||||
000.032s l0025 - check wait time
|
||||
000.046s l0026 - check sleep time
|
||||
000.049s l0027 - check sleep prev time
|
||||
000.053s l0028 - check begin time
|
||||
000.259s l0034 - lower range check
|
||||
000.262s l0035 - upper range check
|
||||
000.265s l0037 - free wait
|
||||
000.270s l0042 - new wait = 1.1 sec
|
||||
000.273s l0043 - check wait time
|
||||
000.276s l0044 - check sleep time
|
||||
000.279s l0045 - check sleep prev time
|
||||
000.282s l0046 - check begin time
|
||||
001.375s l0052 - lower range check
|
||||
001.379s l0053 - upper range check
|
||||
001.382s l0055 - free wait
|
||||
|
||||
TESTS COMPLETED SUCCESSFULLY
|
||||
|
||||
2019-10-30 10:16:51.145 P00 INFO: P1-T1/1 - vm=none, module=common, test=wait (3.87s)
|
||||
2019-10-30 10:16:51.209 P00 INFO: writing C coverage report
|
||||
2019-10-30 10:16:51.674 P00 INFO: TESTS COMPLETED SUCCESSFULLY (5s)
|
||||
```
|
||||
|
||||
An entire module can be run by using only the `--module` option.
|
||||
|
||||
pgbackrest-dev => Run a module
|
||||
```
|
||||
pgbackrest/test/test.pl --vm=none --dev --module=postgres
|
||||
|
||||
--- output ---
|
||||
|
||||
2019-10-30 10:16:52.449 P00 INFO: test begin - log level info
|
||||
2019-10-30 10:16:52.979 P00 INFO: check code autogenerate
|
||||
2019-10-30 10:16:52.979 P00 INFO: cleanup old data
|
||||
2019-10-30 10:16:53.513 P00 INFO: builds required: none
|
||||
2019-10-30 10:16:53.517 P00 INFO: 3 tests selected
|
||||
|
||||
2019-10-30 10:16:57.501 P00 INFO: P1-T1/3 - vm=none, module=postgres, test=client (3.99s)
|
||||
2019-10-30 10:17:00.957 P00 INFO: P1-T2/3 - vm=none, module=postgres, test=interface (3.46s)
|
||||
2019-10-30 10:17:03.325 P00 INFO: P1-T3/3 - vm=none, module=postgres, test=page-checksum (2.37s)
|
||||
2019-10-30 10:17:03.382 P00 INFO: writing C coverage report
|
||||
2019-10-30 10:17:03.944 P00 INFO: TESTS COMPLETED SUCCESSFULLY (11s)
|
||||
```
|
||||
|
||||
### With Docker
|
||||
|
||||
Build a container to run tests. The vm must be pre-configured but a variety are available. The vm names are all three character abbreviations, e.g. `u19` for Ubuntu 19.04.
|
||||
|
||||
pgbackrest-dev => Build a VM
|
||||
```
|
||||
pgbackrest/test/test.pl --vm-build --vm=u19
|
||||
|
||||
--- output ---
|
||||
|
||||
2019-10-30 10:17:04.779 P00 INFO: test begin - log level info
|
||||
2019-10-30 10:17:04.812 P00 INFO: Using cached pgbackrest/test:u19-base-20191012A image (32d78710c7f60872ee4a8b2374d0110e015e2c24) ...
|
||||
2019-10-30 10:17:05.225 P00 INFO: Building pgbackrest/test:u19-build image ...
|
||||
2019-10-30 10:17:05.641 P00 INFO: Building pgbackrest/test:u19-test image ...
|
||||
2019-10-30 10:17:06.013 P00 INFO: Build Complete
|
||||
```
|
||||
|
||||
pgbackrest-dev => Run a Test
|
||||
```
|
||||
pgbackrest/test/test.pl --vm=u19 --dev --module=mock --test=archive --run=2
|
||||
|
||||
--- output ---
|
||||
|
||||
2019-10-30 10:17:06.768 P00 INFO: test begin - log level info
|
||||
2019-10-30 10:17:07.346 P00 INFO: check code autogenerate
|
||||
2019-10-30 10:17:07.346 P00 INFO: cleanup old data and containers
|
||||
2019-10-30 10:17:08.403 P00 INFO: builds required: bin, libc host
|
||||
2019-10-30 10:17:08.615 P00 INFO: 1 test selected
|
||||
|
||||
2019-10-30 10:17:41.992 P00 INFO: P1-T1/1 - vm=u19, module=mock, test=archive, run=2 (33.38s)
|
||||
2019-10-30 10:17:42.163 P00 INFO: no code modules had all tests run required for coverage
|
||||
2019-10-30 10:17:42.163 P00 INFO: writing C coverage report
|
||||
2019-10-30 10:17:42.224 P00 INFO: TESTS COMPLETED SUCCESSFULLY (36s)
|
||||
```
|
||||
|
||||
## Adding an Option
|
||||
|
||||
Options can be added to a command or multiple commands. Options can be configuration file only, command-line only or valid for both. Once an option is added, `config.auto.*`, `define.auto.*` and `parse.auto.*` files will automatically be generated by the build system.
|
||||
|
||||
@ -112,7 +112,8 @@
|
||||
<render type="markdown">
|
||||
<render-source key="index" file="../../../README.md"/>
|
||||
<render-source key="coding" file="../../../CODING.md"/>
|
||||
<render-source key="contributing" file="../../../CONTRIBUTING.md"/>
|
||||
<!-- This needs to be uncommented when generating the contributing doc. We are currently unable to get Docker in Docker running correctly in CI. -->
|
||||
<!-- <render-source key="contributing" file="../../../CONTRIBUTING.md"/> -->
|
||||
<render-source key="test" file="../../../test/README.md"/>
|
||||
<render-source key="documentation" file="../../README.md"/>
|
||||
</render>
|
||||
|
||||
@ -1,14 +1,178 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE doc SYSTEM "doc.dtd">
|
||||
<doc title="{[project]}" subtitle="Contributing to {[project]}" toc="y">
|
||||
<doc title="{[project]}" subtitle="Contributing to {[project]}" toc="y" cmd-line-len="132">
|
||||
<description>{[project]} Contributing Guidelines.</description>
|
||||
|
||||
<!-- Variables used by the rest of the script ============================================================================== -->
|
||||
<variable-list>
|
||||
<variable key="host-contrib-id">contrib</variable>
|
||||
<variable key="host-contrib">pgbackrest-dev</variable>
|
||||
<variable key="host-contrib-user">{[host-user]}</variable>
|
||||
<variable key="host-contrib-image">pgbackrest/doc:contrib</variable>
|
||||
|
||||
<variable key="cwd" eval="y">use Cwd qw(cwd); cwd()</variable>
|
||||
</variable-list>
|
||||
|
||||
<!-- Setup hosts used to build the documentation =========================================================================== -->
|
||||
<host-define if="{[os-type-is-debian]}" image="{[host-contrib-image]}" from="ubuntu:19.04">
|
||||
{[copy-ca-cert]}
|
||||
|
||||
# Fix root tty
|
||||
RUN sed -i 's/^mesg n/tty -s \&\& mesg n/g' /root/.profile && \
|
||||
|
||||
# Suppress dpkg interactive output
|
||||
rm /etc/apt/apt.conf.d/70debconf
|
||||
|
||||
# Install base packages
|
||||
RUN apt-get update && \
|
||||
apt-get install -y sudo ssh curl vim 2>&1
|
||||
|
||||
# Add test user with sudo privileges
|
||||
RUN adduser --disabled-password --gecos "" {[host-user]} && \
|
||||
echo '%{[host-user]} ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
|
||||
|
||||
WORKDIR /home/{[host-user]}
|
||||
|
||||
ENTRYPOINT service ssh restart && bash
|
||||
</host-define>
|
||||
|
||||
<!-- ======================================================================================================================= -->
|
||||
<section id="introduction">
|
||||
<title>Introduction</title>
|
||||
|
||||
<p>This documentation is intended to assist contributors to <backrest/> by outlining some basic steps and guidelines for contributing to the project. Coding standards to follow are defined in <link url="{[github-url-master]}/CODING.md">CODING.md</link>. At a minimum, unit tests must be written and run and the documentation generated before submitting a Pull Request; see the <link section="/testing">Testing</link> section below for details.</p>
|
||||
</section>
|
||||
|
||||
<!-- ======================================================================================================================= -->
|
||||
<section id="environment">
|
||||
<title>Building a Development Environment</title>
|
||||
|
||||
<p>This example is based on <proper>Ubuntu 19.04</proper>, but it should work on many versions of <proper>Debian</proper> and <proper>Ubuntu</proper>.</p>
|
||||
|
||||
<host-add id="{[host-contrib-id]}" name="{[host-contrib]}" user="{[host-contrib-user]}" image="{[host-contrib-image]}" os="u19" option="-v /var/run/docker.sock:/var/run/docker.sock -v {[cwd]}/test:{[cwd]}/test" mount="{[host-repo-path]}:/home/{[host-contrib-user]}/pgbackrest"/>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Install development tools</title>
|
||||
|
||||
<execute user="root" pre="y">
|
||||
<exe-cmd>
|
||||
apt-get install rsync git devscripts build-essential valgrind autoconf
|
||||
libssl-dev zlib1g-dev libperl-dev libxml2-dev libpq-dev libxml-checker-perl
|
||||
libyaml-libyaml-perl libdbd-pg-perl
|
||||
</exe-cmd>
|
||||
<exe-cmd-extra>-y 2>&1</exe-cmd-extra>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<p>Coverage testing is an important component of <backrest/> testing and is integrated directly into the test harness. Unfortunately, the default version of <proper>lcov</proper> is often not compatible with <proper>gcc</proper>. <proper>lcov 1.14</proper> works up to <proper>gcc 8</proper>.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Build lcov 2.14</title>
|
||||
|
||||
<execute pre="y">
|
||||
<exe-cmd>
|
||||
curl -fsSL
|
||||
https://github.com/linux-test-project/lcov/releases/download/v1.14/lcov-1.14.tar.gz |
|
||||
tar zx
|
||||
</exe-cmd>
|
||||
</execute>
|
||||
|
||||
<execute user="root" pre="y">
|
||||
<exe-cmd>make -C lcov-1.14 install</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<p>Some unit tests and all the integration test require <proper>Docker</proper>. Running in containers allows us to simulate multiple hosts, test on different distributions and versions of <postgres/>, and use sudo without affecting the host system.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Install Docker</title>
|
||||
|
||||
<execute pre="y">
|
||||
<exe-cmd>curl -fsSL https://get.docker.com | sudo sh</exe-cmd>
|
||||
<exe-cmd-extra>2>&1</exe-cmd-extra>
|
||||
</execute>
|
||||
|
||||
<execute user="root">
|
||||
<exe-cmd>
|
||||
usermod -aG docker `whoami`
|
||||
</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<p>This clone of the <backrest/> repository is sufficient for experimentation. For development, create a fork and clone that instead.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Clone <backrest/> repository</title>
|
||||
|
||||
<execute err-suppress="y">
|
||||
<exe-cmd>
|
||||
git clone https://github.com/pgbackrest/pgbackrest.git
|
||||
</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
</section>
|
||||
|
||||
<section id="running">
|
||||
<title>Running Tests</title>
|
||||
|
||||
<section id="without-docker">
|
||||
<title>Without Docker</title>
|
||||
|
||||
<p>If <proper>Docker</proper> is not installed, then the available tests can be listed using <setting>--vm-none</setting>.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>List tests that don't require a container</title>
|
||||
|
||||
<execute output="y">
|
||||
<exe-cmd>pgbackrest/test/test.pl --vm=none --dry-run</exe-cmd>
|
||||
<exe-highlight>[0-9]+ tests selected|DRY RUN COMPLETED SUCCESSFULLY</exe-highlight>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<p>Once a test has been selected it can be run by specifying the module and test. The <setting>--dev</setting> option sets several flags that are appropriate for development rather than test. Most importantly, it reuses object files from the previous test run to speed testing. The <setting>--vm-out</setting> option displays the test output.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Run a test</title>
|
||||
|
||||
<execute output="y">
|
||||
<exe-cmd>pgbackrest/test/test.pl --vm=none --dev --vm-out --module=common --test=wait</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<p>An entire module can be run by using only the <setting>--module</setting> option.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Run a module</title>
|
||||
|
||||
<execute output="y">
|
||||
<exe-cmd>pgbackrest/test/test.pl --vm=none --dev --module=postgres</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
</section>
|
||||
|
||||
<section id="with-docker" depend="/environment">
|
||||
<title>With Docker</title>
|
||||
|
||||
<p>Build a container to run tests. The vm must be pre-configured but a variety are available. The vm names are all three character abbreviations, e.g. <id>u19</id> for <proper>Ubuntu 19.04</proper>.</p>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Build a VM</title>
|
||||
|
||||
<execute output="y">
|
||||
<exe-cmd>pgbackrest/test/test.pl --vm-build --vm=u19</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
|
||||
<execute-list host="{[host-contrib]}">
|
||||
<title>Run a Test</title>
|
||||
|
||||
<execute output="y">
|
||||
<exe-cmd>pgbackrest/test/test.pl {[dash]}-vm=u19 {[dash]}-dev {[dash]}-module=mock {[dash]}-test=archive {[dash]}-run=2</exe-cmd>
|
||||
</execute>
|
||||
</execute-list>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section id="option">
|
||||
<title>Adding an Option</title>
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user