db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
77c721eb635c6d757d2d5ed6f2ffc083720c10a7
pgbackrest/src/postgres/interface.h

194 lines
11 KiB
C
Raw Normal View History

Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
/***********************************************************************************************************************************
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
PostgreSQL Interface
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
***********************************************************************************************************************************/
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
#ifndef POSTGRES_INTERFACE_H
#define POSTGRES_INTERFACE_H
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
#include <stdint.h>
#include <sys/types.h>
Add missing header.
2019-12-02 11:54:45 -05:00
#include "common/debug.h"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
#include "common/type/string.h"
Require storage when calling pgControlFromFile(). Previously storageLocal() was being used internally but loading pg_control from remote storage is often required. Also, storagePg() is more appropriate than storageLocal() for all current usage. Contributed by Cynthia Shang.
2019-08-21 11:29:30 -04:00
#include "storage/storage.h"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
/***********************************************************************************************************************************
Defines for various Postgres paths and files
***********************************************************************************************************************************/
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_FILE_BACKUPLABEL "backup_label"
#define PG_FILE_BACKUPLABELOLD "backup_label.old"
Ignore backup_manifest in PG13. This file is created by pg_basebackup so might be in the data directory if the cluster was restored from a pg_basebackup backup. Also exclude backup_manifest.tmp since it is possible to find that in the backup directory.
2020-09-14 10:15:40 -04:00
#define PG_FILE_BACKUPMANIFEST "backup_manifest"
#define PG_FILE_BACKUPMANIFEST_TMP "backup_manifest.tmp"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
#define PG_FILE_PGCONTROL "pg_control"
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_FILE_PGFILENODEMAP "pg_filenode.map"
#define PG_FILE_PGINTERNALINIT "pg_internal.init"
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
#define PG_FILE_PGVERSION "PG_VERSION"
STRING_DECLARE(PG_FILE_PGVERSION_STR);
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
#define PG_FILE_POSTGRESQLAUTOCONF "postgresql.auto.conf"
STRING_DECLARE(PG_FILE_POSTGRESQLAUTOCONF_STR);
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_FILE_POSTGRESQLAUTOCONFTMP "postgresql.auto.conf.tmp"
Rename PostgreSQL pid file constants and tests.
2021-10-13 19:36:59 -04:00
#define PG_FILE_POSTMTROPTS "postmas""ter.opts"
#define PG_FILE_POSTMTRPID "postmas""ter.pid"
STRING_DECLARE(PG_FILE_POSTMTRPID_STR);
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
#define PG_FILE_RECOVERYCONF "recovery.conf"
STRING_DECLARE(PG_FILE_RECOVERYCONF_STR);
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_FILE_RECOVERYDONE "recovery.done"
STRING_DECLARE(PG_FILE_RECOVERYDONE_STR);
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
#define PG_FILE_RECOVERYSIGNAL "recovery.signal"
STRING_DECLARE(PG_FILE_RECOVERYSIGNAL_STR);
#define PG_FILE_STANDBYSIGNAL "standby.signal"
STRING_DECLARE(PG_FILE_STANDBYSIGNAL_STR);
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
#define PG_FILE_TABLESPACEMAP "tablespace_map"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#define PG_PATH_ARCHIVE_STATUS "archive_status"
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
#define PG_PATH_BASE "base"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
#define PG_PATH_GLOBAL "global"
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
STRING_DECLARE(PG_PATH_GLOBAL_STR);
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_PATH_PGMULTIXACT "pg_multixact"
#define PG_PATH_PGDYNSHMEM "pg_dynshmem"
#define PG_PATH_PGNOTIFY "pg_notify"
#define PG_PATH_PGREPLSLOT "pg_replslot"
#define PG_PATH_PGSERIAL "pg_serial"
#define PG_PATH_PGSNAPSHOTS "pg_snapshots"
#define PG_PATH_PGSTATTMP "pg_stat_tmp"
#define PG_PATH_PGSUBTRANS "pg_subtrans"
Add constant for pg_tblspc path.
2019-12-07 17:42:42 -05:00
#define PG_PATH_PGTBLSPC "pg_tblspc"
Add PostgreSQL file and path names used for building manifests.
2019-12-01 16:28:33 -05:00
#define PG_PREFIX_PGSQLTMP "pgsql_tmp"
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
Add Db object to encapsulate PostgreSQL queries and commands. Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check. Add the dbGet() helper function to get Db objects easily. Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together. Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
#define PG_NAME_WAL "wal"
STRING_DECLARE(PG_NAME_WAL_STR);
#define PG_NAME_XLOG "xlog"
STRING_DECLARE(PG_NAME_XLOG_STR);
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
/***********************************************************************************************************************************
Define default page size
Page size can only be changed at compile time and is not known to be well-tested, so only the default page size is supported.
***********************************************************************************************************************************/
#define PG_PAGE_SIZE_DEFAULT ((unsigned int)(8 * 1024))
/***********************************************************************************************************************************
Define default segment size and pages per segment
Segment size can only be changed at compile time and is not known to be well-tested, so only the default segment size is supported.
***********************************************************************************************************************************/
#define PG_SEGMENT_SIZE_DEFAULT ((unsigned int)(1 * 1024 * 1024 * 1024))
#define PG_SEGMENT_PAGE_DEFAULT (PG_SEGMENT_SIZE_DEFAULT / PG_PAGE_SIZE_DEFAULT)
Move WAL constants to postgres/interface header.
2020-09-08 12:44:08 -04:00
/***********************************************************************************************************************************
WAL header size. It doesn't seem worth tracking the exact size of the WAL header across versions of PostgreSQL so just set it to
something far larger needed but <= the minimum read size on just about any system.
***********************************************************************************************************************************/
#define PG_WAL_HEADER_SIZE ((unsigned int)(512))
/***********************************************************************************************************************************
Define default wal segment size
Before PostgreSQL 11 WAL segment size could only be changed at compile time and is not known to be well-tested, so only the default
WAL segment size is supported for versions below 11.
***********************************************************************************************************************************/
#define PG_WAL_SEGMENT_SIZE_DEFAULT ((unsigned int)(16 * 1024 * 1024))
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
/***********************************************************************************************************************************
PostgreSQL Control File Info
***********************************************************************************************************************************/
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
typedef struct PgControl
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
{
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
unsigned int version;
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
uint64_t systemId;
Improve PostgreSQL version identification. Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL. Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required. Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
2020-09-18 16:55:26 -04:00
unsigned int catalogVersion;
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
Add timeline and checkpoint checks to backup. Add the following checks: * Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match. * Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power... * Standby is on the same timeline as the primary. If not, this standby is not following the primary. * Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary. This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup. The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.
2021-12-07 09:21:07 -05:00
uint64_t checkpoint; // Last checkpoint LSN
uint32_t timeline; // Current timeline
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
unsigned int pageSize;
unsigned int walSegmentSize;
bool pageChecksum;
} PgControl;
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
Add WAL info to PostgreSQL interface. This allows the WAL header to be read for any supported version on PostgreSQL.
2019-03-19 19:44:06 +04:00
/***********************************************************************************************************************************
PostgreSQL WAL Info
***********************************************************************************************************************************/
typedef struct PgWal
{
unsigned int version;
Read segment size from WAL headers. This allows validation of the WAL segment size for PostgreSQL versions <= 10.
2020-07-09 17:32:36 -04:00
unsigned int size;
Add WAL info to PostgreSQL interface. This allows the WAL header to be read for any supported version on PostgreSQL.
2019-03-19 19:44:06 +04:00
uint64_t systemId;
} PgWal;
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
/***********************************************************************************************************************************
Functions
***********************************************************************************************************************************/
Remove dependency on pg_database.datlastsysoid. This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column. Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available. The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.
2022-05-04 08:22:45 -04:00
// Is this a template database?
bool pgDbIsTemplate(const String *name);
// Is this a system database, i.e. template or postgres?
bool pgDbIsSystem(const String *name);
// Does this database have a system id, i.e. less than the minimum assignable user id?
bool pgDbIsSystemId(unsigned int id);
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
// Get info from pg_control
Remove pgPath parameter from pgControlFromFile(). In practice this function is always used with storagePg*() so pgPath is known.
2019-10-03 11:14:22 -04:00
PgControl pgControlFromFile(const Storage *storage);
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
// Get the control version for a PostgreSQL version
Remove most references to PostgreSQL control and catalog versions. The control and catalog versions were stored a variety of places in the optimistic hope that they would be useful. In fact they never were. We can't remove them from the backup.info and backup.manifest files due to backwards compatibility concerns, but we can at least avoid loading and storing them in C structures. Add functions to the PostgreSQL interface which will return the control and catalog versions for any supported version of PostgreSQL to allow backwards compatibility for backup.info and backup.manifest. These functions will be useful in other ways, e.g. generating the tablespace identifier in PostgreSQL >= 9.0.
2019-09-07 18:04:39 -04:00
uint32_t pgControlVersion(unsigned int pgVersion);
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
// Convert version string to version number and vice versa
Info objects now parse JSON and use specified storage. Use JSON code now that it is available and remove temporary hacks used to get things working initially. Use passed storage objects rather than using storageLocal(). All storage objects in C are still local but this won't always be the case. Also, move Postgres version conversion functions to postgres/info.c since they have no dependency on the info objects and will likely be useful elsewhere.
2018-09-06 10:12:14 -07:00
unsigned int pgVersionFromStr(const String *version);
String *pgVersionToStr(unsigned int version);
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
// Get info from WAL header
Add storage parameter to pgWalFromFile().
2020-07-21 16:28:05 -04:00
PgWal pgWalFromFile(const String *walFile, const Storage *storage);
Add WAL info to PostgreSQL interface. This allows the WAL header to be read for any supported version on PostgreSQL.
2019-03-19 19:44:06 +04:00
PgWal pgWalFromBuffer(const Buffer *walBuffer);
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
// Get the tablespace identifier used to distinguish versions in a tablespace directory, e.g. PG_15_202209061
Improve PostgreSQL version identification. Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL. Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required. Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
2020-09-18 16:55:26 -04:00
String *pgTablespaceId(unsigned int pgVersion, unsigned int pgCatalogVersion);
Add function to generate PostgreSQL tablespace identifier. In PostgreSQL >= 9.0 each tablespace data is stored in a specially named directory so different major versions can share the same tablespace path.
2019-09-08 06:53:23 -04:00
Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment().
2019-12-08 14:19:47 -05:00
// Convert a string to an lsn and vice versa
uint64_t pgLsnFromStr(const String *lsn);
String *pgLsnToStr(uint64_t lsn);
Add pgLsnFromWalSegment(). Provides the reverse operation for pgLsnToWalSegment().
2020-07-08 12:25:39 -04:00
// Convert a timeline and lsn to a wal segment and vice versa
Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment().
2019-12-08 14:19:47 -05:00
String *pgLsnToWalSegment(uint32_t timeline, uint64_t lsn, unsigned int walSegmentSize);
Add pgLsnFromWalSegment(). Provides the reverse operation for pgLsnToWalSegment().
2020-07-08 12:25:39 -04:00
uint64_t pgLsnFromWalSegment(const String *walSegment, unsigned int walSegmentSize);
Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment().
2019-12-08 14:19:47 -05:00
Add timeline and checkpoint checks to backup. Add the following checks: * Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match. * Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power... * Standby is on the same timeline as the primary. If not, this standby is not following the primary. * Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary. This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup. The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.
2021-12-07 09:21:07 -05:00
// Get timeline from WAL segment name
uint32_t pgTimelineFromWalSegment(const String *walSegment);
Add pgLsnRangeToWalSegmentList() to convert lsn range to wal segments.
2019-12-12 16:43:34 -05:00
// Convert a timeline and lsn range to a list of wal segments
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
StringList *pgLsnRangeToWalSegmentList(uint32_t timeline, uint64_t lsnStart, uint64_t lsnStop, unsigned int walSegmentSize);
Add pgLsnRangeToWalSegmentList() to convert lsn range to wal segments.
2019-12-12 16:43:34 -05:00
Add pgLsnName() and pgXactPath().
2019-12-01 15:49:34 -05:00
// Get name used for lsn in functions (this was changed in PostgreSQL 10 for consistency since lots of names were changing)
const String *pgLsnName(unsigned int pgVersion);
Use stock PostgreSQL page checksum implementation. We were using a customized version which worked fine but was hard to merge with upstream changes. Now this code is maintained much like the types in static.auto.h that we copy and check with each release. The goal is to eventually build directly against PostgreSQL (either source or libcommon) and this brings us one step closer.
2020-03-05 14:23:01 -05:00
// Calculate the checksum for a page. Page cannot be const because the page header is temporarily modified during processing.
uint16_t pgPageChecksum(unsigned char *page, uint32_t blockNo);
Add Db object to encapsulate PostgreSQL queries and commands. Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check. Add the dbGet() helper function to get Db objects easily. Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together. Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
const String *pgWalName(unsigned int pgVersion);
Add pgWalPath() to return version-specific WAL path. Also update the manifest module to use the new function.
2019-12-12 16:11:09 -05:00
// Get wal path (this was changed in PostgreSQL 10 to avoid including "log" in the name)
const String *pgWalPath(unsigned int pgVersion);
Add pgLsnName() and pgXactPath().
2019-12-01 15:49:34 -05:00
// Get transaction commit log path (this was changed in PostgreSQL 10 to avoid including "log" in the name)
const String *pgXactPath(unsigned int pgVersion);
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
/***********************************************************************************************************************************
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Macros for function logging
***********************************************************************************************************************************/
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
String *pgControlToLog(const PgControl *pgControl);
Add WAL info to PostgreSQL interface. This allows the WAL header to be read for any supported version on PostgreSQL.
2019-03-19 19:44:06 +04:00
String *pgWalToLog(const PgWal *pgWal);
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
#define FUNCTION_LOG_PG_CONTROL_TYPE \
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
PgControl
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
#define FUNCTION_LOG_PG_CONTROL_FORMAT(value, buffer, bufferSize) \
FUNCTION_LOG_STRING_OBJECT_FORMAT(&value, pgControlToLog, buffer, bufferSize)
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Add WAL info to PostgreSQL interface. This allows the WAL header to be read for any supported version on PostgreSQL.
2019-03-19 19:44:06 +04:00
#define FUNCTION_LOG_PG_WAL_TYPE \
PgWal
#define FUNCTION_LOG_PG_WAL_FORMAT(value, buffer, bufferSize) \
FUNCTION_LOG_STRING_OBJECT_FORMAT(&value, pgWalToLog, buffer, bufferSize)
Add pgControlInfo() to read pg_control and determine the PostgreSQL version.
2018-04-29 11:20:51 -04:00
#endif
Reference in New Issue Copy Permalink