Maximum Availability Architecture (MAA) with Oracle GoldenGate MicroServices in HUB Architecture: Learn how to use Oracle GoldenGate to improve the availability, reliability, and scalability of your mission-critical systems (English Edition)

By Lucia Hustatyova

Oracle GoldenGate is a software product that allows you to replicate data from one database to another. If you are interested in using Oracle GoldenGate Microservices (OGG MA) in HUB configuration for database upgrade, migration, and data replication, then this book is a valuable resource for you.

This book provides a step-by-step guide for designing and implementing an Oracle GoldenGate Microservices architecture in a hub configuration, with high availability (HA) and disaster recovery (DR) capabilities. It begins by explaining the architecture of Oracle GoldenGate Microservices, and then provides real-world examples of how it can be used to upgrade or migrate databases, replicate data, and monitor the environment. Moving on, the book includes detailed instructions on how to install and configure Oracle GoldenGate Microservices, as well as, how to use the REST API, command line, and web interface to manage the solution. Lastly, the book provides a practical example of how the architecture can be used to achieve HA and DR for data replication.

By the end of the book, you will be a confident and knowledgeable OGG MA user, who is able to design, implement, and manage Oracle GoldenGate Microservices Architecture solutions for your organization.

• Language: English
• Publisher: BPB Online LLP
• Release date: Sep 25, 2023
• ISBN: 9789355515902

Book preview

C

HAPTER

1

Introduction to Oracle GoldenGate in HUB Architecture

Introduction

Oracle GoldenGate Microservices Architecture (OGG MA) replicates data from committed transactions between heterogeneous sources and targets without impacting system performance.

Oracle GoldenGate (MA) is a new microservices architecture that provides REST-enabled services as part of the Oracle GoldenGate environment. The REST-enabled services provide remote configuration, administration, and monitoring through a user-friendly environment (web interface), command line, and APIs.

It provides business continuity by replicating data, that is up to 6 times faster than traditional data movement solutions for high availability, disaster recovery, and zero downtime migrations.

OGG is capable of covering various topologies, as shown in the following Figure 1.1:

One to one (Unidirectional)

One to many (Distribution)

Many to one (Consolidation)

Many to many

Peer-to-peer

Cascading

Bidirectional

Figure 1.1: OGG replication technologies

OGG MA can be used in both homogenous and heterogeneous environments, and it also includes character set conversion capabilities. OGG MA is very useful for multiple use cases:

Data replication between databases

Database upgrades

Database migration (from on-prem to on-prem, from on-prem to Cloud)

Data synchronization

High availability

Using OGG MA, we can achieve zero downtime for database upgrades or migration.

Using the Oracle Veridata tool, we can be sure, that data are 1:1 replicated. With this tool, we can set up multiple jobs to compare data inside the database. We can also see possible inconsistencies in data and then repair and fix reported issues/inconsistencies. Veridata is a Java-based application that runs within the Oracle WebLogic server.

Structure

In this chapter, we will go over the following topics:

Minimal hardware requirements for the OGG MA HUB server

Software requirements for the OGG MA HUB server

Oracle GoldenGate Microservices in HUB architecture

Service Manager

Administration server

Performance Metrics server

Distribution server

Receiver server

OGG MA limitations and restrictions

Oracle Real Application Clusters

Oracle Clusterware

Oracle Grid Infrastructure Agents (XAG)

Oracle Database Files System (DBFS)

Oracle Advances Cluster File System (ACFS)

Objectives

This chapter gives step-by-step instructions on how to install the whole Oracle GoldenGate Microservices Architecture in the HUB configuration. It means that nothing will be installed on the database servers (as a source and target database system/servers). The reason is not only to offload the source and target server and consume server resources but to manage all migrations or replications from one place. Everything can also be monitored via a plugin in Cloud Control. There is a possibility to see reports and parameter files and restart the extract or replicat process.

For this purpose, a separate server will be installed with all required software:

Oracle GoldenGate Microservice architecture (OGG MA)

Binaries are required for all Oracle database versions, which we want to cover for replication or upgrade of the database. (Downloaded will be only one latest version of the OGG MA software, but binaries will be installed for each Oracle database version).

Oracle database: The database is required as a repository for Veridata.

Oracle Weblogic server

Oracle Veridata server

For a secure connection, a self-signed certificate will be used (in the production, it is recommended to use a real certificate signed by the certification authority).

Minimal hardware requirements for OGG MA HUB server

Hardware (HW) requirements are only scaled for testing purposes. In the real configuration, of course, it is advised to use a more powerful setup in the real scenario. Refer to the following Table 1.1:

Table 1.1: Minimal hardware requirements for OGG MA HUB server

Software requirements for OGG MA HUB server

The software requirements for OGG MA HUB server are given in the following Table 1.2:

Table 1.2: Minimal software requirements for OGG MA HUB server

Oracle GoldenGate Microservices in HUB Architecture

A simple view on the OGG MA Hub configuration is as follows:

In the middle is OGG Hub server with all required software.

Veridata software and database can be installed on different servers or be part of the OGG MA Hub server, but in this case, you need to have a very powerful server as this will consume a lot of server resources.

On the left side is a database server/database from where data will be extracted.

On the right side is a database server/database where data will be replicated.

Versions of the databases can be different; the character set can be different, but the character set of the target database must be a superset of the character set of the source database, and the architecture type can be different (classic or multitenant architecture).

Connection from the OGG MA Hub server to database servers/databases is via Oracle clients installed on the OGG MA Hub server and is based on the TNS entries in the Oracle client.

In the end, it means that really nothing needs to be installed on the source and target database server. The database (as a target) where data will be replicated must be prepared in advance.

We can monitor the replication process directly in the OGG MA Hub server (via the Service Manager web portal) or via Oracle Cloud Control after installing a plugin to cover OGG. In Service Manager, you can also see a lot of statistics in the section Performance Metrics Server.

When replication is finished, we can compare data between source and target via Oracle Veridata. In the Veridata application, it is possible to create multiple jobs (based on the table, schema, or whole database) to verify consistency. OGG MA does not guarantee data consistency even if we do not see any errors during the extract or replicate process. Therefore, it is recommended to use an additional tool provided by Oracle – Oracle Veridata or another third-party tool or just own scripts for comparisons. With Veridata, there are some benefits –you can see exactly which rows are not consistent, compare the content of the rows, and run a fix-up/repair job. Refer to the following Figure 1.2:

Figure 1.2: High-Level Design of the Oracle GoldenGate HUB Server

OGG MA consists of the following main components/servers:

Administration server

Performance Metrics server

Distribution server

Receiver server

Service Manager

Additionally, Oracle GoldenGate MA includes two more components:

Admin client: Similar to the ggsci in classic architecture

Microservice Security, Authentication/Authorization services

Service Manager

The Service Manager is the main Web User Interface in which we can manage all deployments in OGG MA infrastructure. The advantage of the HUB configuration is that everything is managed from this one place. The Service Manager is running as a system service, and via the Service Manager portal, we manage all deployments.

The Service Manager is installed together with the installation of the first OGG deployment. It is not possible to install it separately. After installation of the first deployment and login to the Service Manager, you can see that each deployment has started/installed an additional four servers:

Administration Server

Distribution Server

Performance Metrics Server

Receiver Server

The first deployment is recommended to create/install for the highest version because with the first deployment, the Service Manager is also created. This means that we will replicate data from Oracle version 11g to 19c; so firstly, we will create a deployment for the 19c database. After the successful installation of the first deployment, the Service Manager is available.

Welcome page when Service Manager is installed (:) is shown in the following Figure 1.3:

Figure 1.3: Oracle GoldenGate Service Manager login page

Each server is running on the dedicated port that was specified during the installation of the deployment. Deployment must be installed for each database – source and target separately, and each deployment will install and start the mentioned servers. So for one replication between source and target database, you need to reserve 10 available ports. It is recommended to check the availability of the ports before installation of the new deployment. You can check available ports with the following command:

/usr/sbin/lsof -i -P -n | grep LISTEN|grep

The Service Manager is installed together with the first deployment; therefore, after login, you can see already started servers for deployment in the following Figure 1.4:

Figure 1.4: Oracle GoldenGate Service Manager - Overview of all services/deployments

Administration server

We can call the Administration server as a supervisor for each deployment. It provides different services for all processes of all registered deployments, active or inactive, such as:

Supervision

Administration

Management

Monitoring

With the administration server, we are able to:

Register/unregister extract/replicat processes.

Create extract/replicat processes.

See the current parameter file and manage it.

See reports and discard files.

Ability to see different types of statistics from the extract/replicate processes.

Monitor processes and see more detailed information about them.

Monitor LAG in replicat process.

Auto-start tasks.

Auto-restart tasks.

Supplemental logging in the source database system.

Manage checkpoints and heartbeat.

Manage user access: Add/delete users who can access the Administration server in Service Manager.

Manage credential store and encryption keys.

The most powerful feature of the Administration server in Oracle GoldenGate MA is the included REST API Service Interface, which can be accessed from any HTTP/HTTPS client. Communication with the Administration server can also be via the admin client from the server to perform REST API calls; for example, manipulate with the extract, see and update parameter files, restart processes, and so on.

The administration server is accessible via the Service Manager web page after clicking on the port in the line mentioned Administration Server. Login credentials are the same as for the Service Manager, as can be seen in the following Figure 1.5:

Figure 1.5: Oracle GoldenGate Administration Server - login page

After entering credentials, the following page shown in Figure 1.6 will be displayed. It is related to the main page with setup extract and replicate processes. On this page, you can also see (later, after configuration) Critical Events.

Figure 1.6: Oracle GoldenGate - Administration Server - main page

Performance Metrics server

In the Performance Metrics server, we can see different types of statistics and metrics from all processes in Oracle GoldenGate, such as extract and replicate process, cache statistics, thread performance, server statistics, or queue statistics. Refer to the following Figure 1.7:

Figure 1.7: Oracle GoldenGate - Performance Metrics server

Distribution server

The distribution server behaves as a process that is responsible for the networked data distribution as an agent. The distribution server is able to transfer and process data that are extracted from the source database system. It is able to handle multiple data streams from multiple trail files in order to target trail files and everything at once. Communication is based on the path, meaning there is a created channel between the source and target systems, and trail files are transferred from the source to target specified location.

In the Distribution server, we can:

Create/delete path for distribution.

Manage existing paths.

Restart (stop/start) paths.

Refer to the following Figure 1.8:

Figure 1.8: Oracle GoldenGate - Distribution path

Distribution server can be configured in OGG MA and also in OGG Classic Architecture (OGG CA). The difference is that in OGG MA, the path is configured between the Distribution and Receiver server. In the combination of OGG MA and OGG CA, the communication is based on the classic processes (collectors).

Compared with Oracle GoldenGate Classic architecture, Microservice architecture replaces old fashion method using data pumps with a single instance service. This means that at once, it will distribute one or more trail files to multiple destinations – multiple Receiver servers.

For the distribution, we can use different communication protocols:

UDT: UDP-based Data Transfer protocol, optimized for high-speed wide area network. This type of protocol ensures an efficient alternative to the TCP protocol, and it is able to transfer huge amounts of data (high-volume transfer of large datasets of data via Wide Area Network, or WAN) at higher speed.

WebSocket (WS) or WebSocket Secure (WWS): WebSocket for HTTP/HTTPS. WS is used in plain HTTP, and WWS relies on the SSL/TLS security used in HTTPS. The WSS protocol ensures that the connection is encrypted, unlike the WS protocol performs an unencrypted connection.

Oracle GoldenGate protocol (OGG): Dedicated to the communication between the Distribution server and the Collector in combination with the classic target (OGG Classic).

Proxy: Especially for the Cloud environment, it supports the following:

CKS5 for any network protocol

TTP for HTTP protocols only (including WebSocket)

Receiver server

The receiver server is responsible for handling incoming trail files. In OGG MA, it communicates with the Distribution server based on the created path. In the Receiver server, it is possible to:

Monitor path

Diagnose and troubleshoot path issue

See different statistics about the path

OGG MA limitations and restrictions

Oracle GoldenGate does not support all datatypes. Please check carefully before starting to migrate, in order to avoid issues further down the line.

Link to Oracle documentation:

https://docs.oracle.com/en/middleware/goldengate/core/19.1/oracle-db/1-understanding-whats-supported.html#GUID-110CD372-2F7E-4262-B8D2-DC0A80422806

For example, some datatypes are not supported:

Tables with restricted uniqueness

Special cases of:

XML types

UDTs

Object tables

Collections or nested tables

Oracle Real Application Clusters

Oracle Real Application Clusters (RAC) provides the ability to run multiple instances which are running on multiple servers in the cluster and they are connecting to one Oracle database. For this type of configuration, we need to have at least two servers in the cluster. Oracle RAC is known as a high-availability solution, such that in case one member in the cluster fails, there is still another member to cover user requests.

In a situation when we need to improve performance with more hardware resources, we can still add additional servers to the cluster without downtime or interrupting users’ access to the database. However, even though Oracle RAC can have multiple nodes, it is still not recommended to add more than 3. This is because RAC uses a Global Cache, and when too many nodes are participating, it can cause slowness for certain operations. Moreover, it is always better to upscale the node. Therefore, add more memory or other resources for existing nodes instead of adding more nodes.

Oracle Clusterware

Oracle Clusterware is a Cluster manager for the Oracle database. For example, in the Oracle RAC configuration, Oracle Clusterware monitors all Oracle resources. If something fails, Oracle Clusterware will try to start it automatically. With clusterware commands we can also relocate resources to another server.

Oracle Grid Infrastructure Agents

Oracle Grid Infrastructure Agents (XAG) is known as a component of Oracle Grid Infrastructure that provides a High Availability Framework to application resources and types which are managed via the agent management interface AGCTL. There is some predefined Oracle Grid Infrastructure resource configuration, for example, for Oracle GoldenGate start or stop processes, and so on.

Oracle Database Files System

An alternative for using classic NFS is to implement the Oracle Database Files System (DBFS), which also looks like a normal local file system attached to the server. It also has two components: server and client.

This functionality helps manipulate the files in the database via any operating system program that works with files.

DBFS will create a standard file system on top of the physical structure (files and directories) that are stored in the database tables, as a DBFS Content Store. In the background, we can manipulate with the files via PL/SQL procedures, for example, create, list, open, read or write.

In the database, there are files stored as Oracle SecureFiles LOBs. This means that the advantage of using the DBFS will profit with all benefits from high availability and recovery options provided by Oracle database. Each database user can create one or more file systems, mount them via client and the content of those filesystem is stored in the database in the dedicated table.

Oracle Advanced Cluster File System

Extension of Oracle Automatic Storage Management (Oracle ASM), which supports all users’ files, is Oracle Advanced Cluster File System (ACFS). It is a more scalable and multi-platform file system integrated directly with the Grid Infrastructure.

With this functionality, we can achieve HA and DR for OGG MA. ACFS will be used to store OGG MA deployments.

Conclusion

After completing this chapter, you should be able to understand the OGG MA infrastructure and the logic of the architecture in the HUB configuration, identify and describe all components and their functionality.

To achieve DR and HA for the whole OGG MA in HUB configuration, the next chapters will explain how to configure and set up a complete environment.

Points to remember

Powerful is Oracle GoldenGate Microservices and where and how it can be used.

It can be used in different ways and the different topologies. The chapter explains in detail each component of Oracle GoldenGate Microservices, especially in HUB architecture.

As the whole OGG HUB configuration will cover High Availability and Disaster Recovery, it is also explained what is Oracle Grid Infrastructure with ACFS.

Question

Explain the difference between the normal installation of OGG MA and OGG MA in HUB infrastructure. List down all main components of the OGG MA. What will be used for sharing files between server nodes in the cluster.

Join our book’s Discord space

Join the book’s Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors:

https://discord.bpbonline.com

C

HAPTER

2

Oracle GoldenGate MicroServices Architecture in HUB Configuration

Introduction

Before starting the installation of the whole infrastructure where a plan is to use HUB configuration, we need to build High Level Design (HLD) for it.

In HLD, we will learn where and what will be installed, and how the flow will go for each use case where Oracle GoldenGate Microservices will be used.

Structure

In this chapter, we will go over the following topics:

High-Level Designs

HLD for database upgrade/migration

♢ Database upgrade/migration from non-multitenant to non-multitenant environment

♢ Database upgrade/migration from non-multitenant to multitenant environment

♢ Database upgrade/migration/convert from non-multitenant to multitenant environment

HLD for data replication

♢ HLD for one-way replication

♢ HLD for bi-directional replication

♢ HLD for multi-target replication

Useful observations

Objectives

This chapter deals with the use cases where Oracle GoldenGate can ensure almost zero downtime for the database and from the application perspective. It only requires reconnection to another database. You will learn about the use cases Upgrade database, Migrate database and Data replication

High Level Designs

Each High-Level Design (HLD) will have an explanation of how the flow/process is going in sequences: graphical interpretation and description for HLD. As OGG MA can be used for multiple purposes, we will explain each use case:

Database upgrade

Database migration

Non-multitenant to non-multitenant

Non-multitenant to multitenant

Character set conversion

Data replication

One-way replication

Bi-directional replication (also known as master/master replication)

Multi-target replication

HLD for database upgrade/migration

The main distinction between the terms upgrade and migrate lies in what is being changed at the end, whether it is the database with the data or the software components.

Upgrade transforms existing environments, such as installed components and Oracle internal views, based on the new Oracle release. However, in the end, user data is not directly affected and touched, and of course, they are not changed during the upgrade process.

On the other side, Migration means that user data is touched and moved somewhere else, for example, to a new server, new storage, or a completely new environment. In this case, it does not mean that we need to change the version of the Oracle software also, but the objective could be to save time and a number of downtimes. For the migration, different methods can be used, for example, using DataGuard, RMAN, OGG, expdp/impdp, or also combination of multiple mentioned methods, that is, everything that can speed up migration process. The goal is to decrease downtime for customer.

This chapter will focus on the migration using OGG and combination with Oracle Data Guard or expdp/impdp in some cases (because initial load can be also performed directly with OGG, but of course with some limitations. Therefore, it is nice to have a good analysis of your environment and data inside the database).

Database upgrade/migration from non-multitenant to non-multitenant environment

When just an upgrade of the database is planned, without any changes, such as the architecture type of the database (non-multitenant to multitenant), or no changes in the character set, it is enough to install a new environment for the upgrade, that can be performed on the same or another server.

In the scenario where the server is also changed, it is not only an upgrade, but it is already a migration of the database, as we will move the whole environment to another server.

Here are two possible approaches:

Database upgrade on the same server: Here, the server is not changed. We need to install a new Oracle software with a new version and an empty database with a different, at least instance name (or also complete renaming of the database can be performed). After finishing the replication, we need to rename it; if there is a requirement to have the same name for the database.

Database upgrade together with migration: In this scenario, we do not need to change a database name, as the completely new environment will be installed on the new server.

In HLD, the flow when OGG MA is used for the upgrade (or design can also be used for migration to another server) is described.

The first HLD describes the flow using OGG MA for the initial load, and the second HLD describes the flow via export/import via Data Pump.

Refer to the following Figure 2.1:

Figure 2.1: Database upgrade/migration from non-multitenant to non-multitenant using OGG MA for initial load

Refer to the following Figure 2.2:

Figure 2.2: Database upgrade/migration from non-multitenant to non-multitenant environment with OGG MA and initial load performed with export/import

Description for use case Oracle upgrade from 12c to 19c (non-multitenant) Prepare environment:

Follow the given steps:

Server is available for the upgrade/migration.

Installation of the new empty environment 19c & apply required patches.

Install new Oracle software: Oracle 19c

Create a new empty Oracle database 19c (non-multitenant)

Patch the Oracle software and database with the latest available patch (release update)

Prepare both databases for the OGG.

Database must be in archive log mode

Supplemental login must be enabled

Force logging must be enabled

OGG parameter (enable_goldengate_replication) must be set to TRUE

Memory parameters:

SGA >= 5G

streams_pool_size>= 1G

open_cursors>= 1000

New database schema for OGG setup will be created with required roles and grants with the dedicated tablespace.

On the OGG HUB server, set up a connection for both databases in the tnsnames.ora located in the client of the OGG MA installation.

Check the connection from the OGG HUB server to the source and target database.

On the OGG HUB server, create a new deployment for the source and target database.

At this point, you should see both deployments (all 4 servers running: Administration Server, Distribution server, Performance metric server, Receiver server) for the source and target database.

On the OGG HUB server in Service Manager, set up credentials.

Check connection via GUI (Service Manager) to both databases.

Start capture process:

Follow the given steps:

On the OGG HUB Server, setup the standard Capture/Extract process (as a first step) from Oracle database 12c.

The process to capture all committed changes (transactions) in the database will start, and it will save them to the trail files located on the OGG HUB server (defined path, the recommendation was to use a separate filesystem for this). The DR and HA environment, for this purpose, will be used ACFS.

Immediately when the process is started, it is required to notice (remember) the SCN of the database.

As soon as the process is started, trail files are generated in the OGG server in the defined destination.

Distribution path: As we are using the OGG HUB server, we do not need to set up a distribution path. However, it can be used when we want them in both places; double space is required for the trail files.

Decision which method will be used for the initial load:

In this step, you need to decide which method you want to use for the initial load of data from the source database to the target database. In summary, you have options:

Using OGG

Using Data Pump

Here, it is very difficult to say which method is the right method for you. Everything is based on your environment and your analyses. In case of an uncomplicated database structure, it is recommended to use an initial load using OGG. In case your environment is very complicated with a lot of objects and different types of objects (for example, Oracle views with more than 2 tables), it is better to use export/import via Data Pump (at least you will meet less complications during the initial load of data to the target system). Moreover, in very complex scenarios, it is faster to perform the initial load using export/import via a Data Pump.

Initial-load using Data Pump

If you have chosen to use the Data Pump method, there is no need for you to perform structural export/import. In case you plan to change the character set for the database, it is recommended to analyze data via the DMU tool, if there are any issues during character set conversion (for example, length of the column, special characters, and so on.) In case of some findings, it is recommended to fix this before starting the initial-load replication process. From the DMU tool, you will see the conflicted data.

The Oracle Database Migration Assistant for Unicode (DMU Tool) can help you identify problematic data after character set migration from any character set to a Unicode character set. With this tool, you can create problem reports and fix data before importing them to the target system. Run the DMU tool if character set conversion is planned.

Follow the given steps:

On the source database, perform structural export of the data using Data Pump.

On the target database, perform structural import of the database also using Data Pump.

Fix findings reported from the DMU tool.

On the target database, perform import of the data using Data Pump. Wait until it is successfully finished.

On the OGG HUB server, configure the Replication process. However, do not start it. It will be started later using the option start from the SCN. Thus, all committed changes in the source database, from the time when the capture process was started until the time when the import data using the Data Pump was successfully finished, will be replicated in the target database.

Initial-load using OGG

If you decide to use OGG for the initial load, you must perform structural export from the source database and import to the target database. Again, if you plan to change the character set of the database, please use the DMU tool to analyze your database data. In case of any findings, fix them before starting the initial-load replication process. Follow the given steps:

Run the DMU tool if character set conversion is planned.

On the source database, perform structural export of the data using Data Pump.

On the target database, perform structural import of the database also using the Data Pump.

As the OGG does not handle referential constraint rules when data are loaded, it is recommended to disable them before starting the replicate/load process. They can be enabled after the initial load is finished.

To speed up activity during the replication process, you can also exclude indexes and recreate them after initial-load replication is finished.

DDL must be disabled in the extraction and replication process. DDL processing is disabled