From 86a651f9b03890af4f79076600e983bba624f8c5 Mon Sep 17 00:00:00 2001 From: David Steele Date: Thu, 12 Aug 2021 07:49:59 -0400 Subject: [PATCH] Add optional stress test section to the user guide. This is intended to provide pre-release stress-testing. Include container memory limits to help check for memory leaks. Also add parallelism to make for faster builds. --- doc/RELEASE.md | 34 +++++ doc/xml/user-guide.xml | 300 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 308 insertions(+), 26 deletions(-) diff --git a/doc/RELEASE.md b/doc/RELEASE.md index cf33aad11..314cf92f4 100644 --- a/doc/RELEASE.md +++ b/doc/RELEASE.md @@ -49,6 +49,40 @@ git commit -m "Release test" git push origin release-ci ``` +## Perform stress testing on release + +- Build the documentation with stress testing enabled: +``` +${PGBR_REPO?}/doc/doc.pl --out=html --include=user-guide --require=/stress --var=stress=y --var=stress-scale-table=100 --var=stress-scale-data=1000 --pre --no-cache +``` + +During data load the archive-push and archive-get processes can be monitored with: +``` +docker exec -it doc-pg-primary tail -f /var/log/pgbackrest/demo-archive-push-async.log +docker exec -it doc-pg-standby tail -f /var/log/pgbackrest/demo-archive-get-async.log +``` + +During backup/restore the processes can be monitored with: +``` +docker exec -it doc-repository tail -f /var/log/pgbackrest/demo-backup.log +docker exec -it doc-pg-standby tail -f /var/log/pgbackrest/demo-restore.log +``` + +Processes can generally be monitored using 'top'. Once `top` is running, press `o` then enter `COMMAND=pgbackrest`. This will filter output to pgbackrest processes + +- Check for many log entries in the `archive-push`/`archive-get` logs to ensure aync archiving was enabled: +``` +docker exec -it doc-pg-primary vi /var/log/pgbackrest/demo-archive-push-async.log +docker exec -it doc-pg-standby vi /var/log/pgbackrest/demo-archive-get-async.log +``` + +- Check the backup log to ensure the correct tables/data were created and backed up. It should look something like: +``` +INFO: full backup size = 14.9GB, file total = 101004 +``` + +- Check the restore log to ensure the correct tables/data were restored. The size and file total should match exactly. + ## Clone web documentation into `doc/site` ``` cd ${PGBR_REPO?}/doc diff --git a/doc/xml/user-guide.xml b/doc/xml/user-guide.xml index d1e7b00bb..cc4dc2e2d 100644 --- a/doc/xml/user-guide.xml +++ b/doc/xml/user-guide.xml @@ -31,6 +31,11 @@ y + + n + 1 + 1 + '{[os-type]}' eq '{[os-debian]}' '{[os-type]}' eq '{[os-rhel]}' @@ -240,26 +245,6 @@ pg_lsclusters systemctl status postgresql-{[pg-version-upgrade]}.service - - - mkdir -p -m 700 /root/.ssh && \ @@ -706,10 +691,10 @@ Introduction - + - +

This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once is up and running then skipping around is possible but following the user guide in order is recommended the first time through.

@@ -803,7 +788,7 @@

{[user-guide-os]} packages for are available from Crunchy Data or yum.postgresql.org, but it is also easy to download the source and install manually.

- +

When building from source it is best to use a build host rather than building on production. Many of the tools required for the build should generally not be installed in production. consists of a single executable so it is easy to copy to a new host once it is built.

@@ -858,6 +843,7 @@ cd /build/pgbackrest-release-{[version]}/src && ./configure && make + -j 4 @@ -868,7 +854,7 @@

A new host named {[host-pg1]} is created to contain the demo cluster and run examples.

- + +
+ Stress Testing + +
+ Configuration + + + Configure {[host-repo1]} for stress testing + + 8 + lz4 + 1 + 1 + y + + + + Create the {[host-pg1]} spool directory + + + mkdir -p -m 750 {[spool-path]} + + + chown postgres:postgres {[spool-path]} + + + + + Configure {[host-pg1]} for stress testing + + 8 + y + lz4 + 1 + {[spool-path]} + y + 4 + 4 + + + + Create the {[host-pg2]} spool directory + + + mkdir -p -m 750 {[spool-path]} + + + chown postgres:postgres {[spool-path]} + + + + + Configure {[host-pg2]} for stress testing + + 8 + y + lz4 + 1 + {[spool-path]} + y + 4 + 4 + +
+ +
+ Create Tables and Load Data + +
+ Break Streaming Replication + +

Break streaming replication to force the standby to replicate from the archive during data load.

+ + + Break streaming replication by changing the replication password + + + + psql -c "alter user replicator password 'bogus'" + + + + + + Restart standby to break connection + + + {[pg-cluster-restart]} + + +
+ +
+ Create Tables + + + Create tables + + + psql -c " + create or replace function create_test_table(prefix int, scale int) returns void as \$\$ + declare + index int; + begin + for index in 1 .. scale loop + execute 'create table test_' || prefix || '_' || index || ' (id int)'; + end loop; + end \$\$ LANGUAGE plpgsql;" + + + + + bash -c 'for i in {1..{[stress-scale-table]}}; + do psql -c "select create_test_table(${i?}, 1000)"; + done' + + + +
+ +
+ Load Data + + + Load data + + + pgbench -n -i -s {[stress-scale-data]} + 2>&1 + + +
+ +
+ Fix Streaming Replication + +

Fix streaming replication so backups will work. Note that streaming replication will not start again until all WAL in the archive has been exhausted.

+ + + Fix streaming replication by changing the replication password + + + + psql -c "alter user replicator password 'jw8s0F4'" + + + +
+
+ +
+ Testing + +
+ Full Backup + + + Full backup + + + + pgbackrest --stanza=demo --type=full + --log-level-console=info --log-level-file=detail backup + + 2>&1 + + +
+ +
+ Diff Backup with Delta + + + Diff backup + + + + pgbackrest --stanza=demo --type=diff --delta + --log-level-console=info --log-level-file=detail backup + + 2>&1 + + +
+ +
+ Restore with Delta + + + Stop <postgres/> + + + {[pg-cluster-stop]} + + + + + Restore + + + + pgbackrest --stanza=demo --type=standby --delta + --log-level-console=info --log-level-file=detail restore + + 2>&1 + + +
+ +
+ Restore + + + Remove data + + + + rm -rf {[pg-path]} + + + + + + Restore + + + + pgbackrest --stanza=demo --type=standby + --log-level-console=info --log-level-file=detail restore + + 2>&1 + + + + + Start <postgres/> + + + {[pg-cluster-start]} + + + + + Check cluster + + + psql -c "select count(*) from pg_class" + + +
+
+
+
Upgrading <postgres/>